Powered By Blogger

2015/01/23

Oracle RAC 11gR2 on OEL6

Let's begin 2015 by trying to detail an Oracle Real Application Cluster 11G Release 2 Installation on Oracle Enterprise Linux 6 (the update used during this install is 6.4 , but this will most probably works with other OEL6 update). 
To make things easier (and more readable), I've divided this post in 04 main sections. The first section is just about Physical Installation, the second is about OS Installation and Pre-Installation Steps, the third one will deal with Grid Infrastructure Software Installation and the Fourth is about RDBMS Software Installation.

I. Physical Installation:

 This very short section is just about the description of Physical Installation performed for this Installation. As a picture is worth a thousands words, let's start by showing a small picture of the Infrastructure we are deploying.




So, we've 02 Physical Nodes with 04 Ethernet Ports for each nodes. Two of these Ethernet Ports will be slaves of Bond0 (active-passive mode) and connected to a Public Network while the others two ports will be initially configured as Independent Interfaces and connected to a Private Network. In fact, Rather than bonding these private NICs, we leave them both configured with static fixed addresses and during the installation, we'll specify both interfaces as private. Grid Infrastructure will assign a virtual IP address in 169.245.x.x network to these Interfaces and use HAIP protocol. The result is equivalent to Bonding in Load-balancing Mode.
For Storage connectivity, we have 02 FC Ports connected on a Storage Area Network, Shared Disks that will be used later as ASM Devices are provided by a Shared Storage connected to this SAN.
To close this section, I have few words about Nodes Physical Requirement: The minimum required RAM is 2.5 GB. For systems with 2.5 GB to 16 GB RAM, Oracle recommends to use swap space equal to RAM. For systems with more than 16 GB RAM, use 0.75 x RAM as the swap space.

 II. OS Installation and Pre-Installation Steps:

 Having Physical Nodes Installation completed, we can move forward with the OS Installation and the Cluster Prerequisites Configuration.

 II.I: OS Installation:

 The Systems Installation is fully completed using Kickstart file.  I won't discuss the many advantages offered by Kickstart Installation, but will just highlight one of the main benefit I've found when using it for a cluster Installation (Not just Oracle RAC, but also RHEL Cluster, Linux-HA and other cluster frameworks). This advantage is the almost perfect symmetry between the clustered nodes (better make sure of that for a cluster Installation). There's also a large number of  configurations that can be completed using the %post section of Kickstart .  For instance, I've populated the /etc/hosts file and configured NTP in the %post section of the Kickstart File shown below.
In fact, I could have automated all the configurations that will be completed in the section II in this kickstart, but I've preferred to leave it like that in order to well describe these configurations in the next sections....


II.2: Network Configuration:
As described in the first Section, we've 02 Physical Nodes with 04 Physical Ethernet Interfaces . For redundancy purpose, the first two network interfaces are slaves of Bond0 and will be used for public IP. Though we've configured the 02 Private Interfaces as Independent Interfaces, HAIP will automatically be configured during the Grid Infrastructure Installation. 

In fact, HAIP is enabled for the private network by selecting multiple interfaces to use as private interfaces during the Grid Infrastructure Installation. This functionality is available starting with Oracle Grid Infrastructure 11g Release 2(11.2.0.2). Though it's enabled by default, we still have to set IP on the interfaces and plumb/up these interfaces. In fact, Oracle Grid currently creates a Virtual  IP on the private network adapters using the 169.254.*.* subnet for the HAIP. The purpose of HAIP is to load balance across all active interconnect interfaces, and failover to other available interfaces if one of the existing private adapters becomes unresponsive.


Each node will need a Virtual IP, and we'll also need a set of 03 Scan IPs. Below is a table with Details about this IP Address scheme.


Node Name
Public IP / Interfaces
Private IP Interface (HAIP)
Public Virtual IP
dbnode01
192.168.0.11
(bond0)
192.168.1.41 (eth2)/
192.168.1.43 (eth3)

192.168.0.15
dbnode02
192.168.0.12
(bond0)
192.168.1.42 (eth2)/
192.168.1.44 (eth3)

192.168.0.16
Scan Name
dbnode-scan.stivesso.local
192.168.0.17
192.168.0.18
192.168.0.19
Public and Private IPs were configured during the System Installation, with Public Interfaces being bonded as bond0 while private Interfaces were configured as Independent Interfaces. Below, we can see the output of the configuration applied by the KickStart configuration on these Interfaces.

 

The Public/Private and VIP with names associated  are all added to the DNS  and to the systems hosts files (see hosts file below, populated during the Kickstart Installation), while The scan IP/names associations must only be added to the DNS.

***/etc/hosts***
 

 ***DNS resolver / Make sure than scan name can be resolved***



The other important Network Service to configure is the NTP as we must make sure that server's clocks are synchronized. For this case I use a set of Internal NTP Servers and modified just the server statement in /etc/ntp.conf (replaced the default NTP Servers addresses by my own)


Also, to make sure that I won't hit PRVF-5436 error related to NTP configuration,I had to add the -x option to ntpd configuration and restart the NTP daemon. That was also completed during the Kickstart Installation. Below is the file that was populated.


Finally, the following Kernel Parameters need to be set on Private Interfaces for all the nodes. That'll disabled the RPF. Reverse Path Filtering (RPF) is a security feature which allow the kernel to throw away packets if the reply of a packet may not go through the interface it was received on. 




 II.3 User Accounts and System Parameters
Let's now complete the creation of required User accounts and Systems parameters setting. Note that Oracle recommends the usage of different users accounts for the installation of the Grid Infrastructure (GI) and the Oracle RDBMS home. Following this recommendation , I'm trying to have Grid Infrastructure owned by user 'grid' and Oracle RDBMS owned by user 'oracle'. 
Most of the pre-installation configuration tasks can be completed by using the Oracle RDBMS Server 11gR2 Pre-install RPM for Oracle Linux 6 x86_64 (64 Bit) architecture, available from the Unbreakable Linux Network, or via the Oracle Public Yum repository. This package replaces the previous Oracle Validated RPM.

The Oracle RDBMS Pre-install RPM does the following:

* Automatically installs any additional packages needed for installing Oracle Grid       Infrastructure and Oracle Database 11gR2 (11.2.0.3).

* Creates an oracle user, and creates the oraInventory (oinstall) and OSDBA (dba) groups for that user. For security purposes, this user has no password by default and cannot login remotely. To enable remote login, please set a password using the "passwd" tool.

* Sets and verifies sysctl.conf settings, system startup parameters, user limits, and driver parameters to the minimum acceptable values based on recommendations from the Oracle Database.


Below is the package Installation procedure details (on both nodes), using Oracle Public Yum repository.



 To complete with software Installation, I recommend to also install cvuqdisk which is coming with the Grid Software, it is located under the rpm folder of the Grid Installer Binaries (see below).


As Oracle recommends different users for the installation of the Grid Infrastructure and the Oracle RDBMS home, the next task is to create the grid Users and required Groups manually. We're also settings the user limits for this newly created Grid User. So on every Cluster Node, let's run the following (on both nodes).
Note that the UID for the Oracle User created by Oracle-rdbms-server-11gR2-preinstall is 54321, so I choose the next UID for Grid (same for the groups)




Now, we've to modify the User Limits settings for grid User (can also adjust if needed). That was done by the installation of the oracle-rdbms-server-11gR2-preinstall Installation for Oracle User, below is  /etc/security/limits.conf after this package's installation.

We're just modifying the file by adding Grid Settings as seen below:

Having the Users/Groups configured, we can move forward with the creation of the required Directory for both Oracle and Grid Users. As seen in my kickstart file, I'm using lv_u01 (mounted on /u01) for Oracle Software Binaries. I'm using the script below to complete the creation of the required directories and setting of their permissions (see comments in the script for details about each directory)

The next task of this sub-section is to reflect the created directory structure to grid and oracle Users environment by setting some of their environment variables. Note that for Oracle User, I'm using the same parameters on all the nodes, while ORACLE_SID for grid user across nodes differs.

Bash Profile for Oracle User on the 02 nodes


 Bash Profile for grid User on Node1


 Bash Profile for grid User on Node2
 

Finally, we need to create password for both Oracle and Grid using passwd command (just showing this as an example, Create your own password in a more secure way:-) )




II.4 Shared Storage Multipath Configuration
As described in Section I, the shared Disks are sitting on an external storage, and are accessible through a Multipath Software. For this configuration, we're making use of DM-Multipath. But before getting to Multipath configuration, let's summarize in a table the Shared Physical LUNs and the role they're supposed to play. 


Share Devices
Multipath Name
Size
Description

asm-ocr-disk1
1GB
Oracle RAC OCR Disk

asm-ocr-disk2
1GB
Oracle RAC OCR Disk

asm-ocr-disk3
1GB
Oracle RAC OCR Disk

asm-data-disk1
500GB
ASM Data Disk

The first three devices are to be added in a dedicated Disk Group for OCR/Vote Disk while the last one will be used for DB Data Disk Group. Before moving with Multipath Configuration, let's just rescan the scsi Devices to make sure that all the allocated LUNs are present on the Systems (to do on all the Cluster Nodes).

We can now start the multipath configuration by using mpathconf to make the initial configuration easy. 
Note that most of the steps below are completed on one node. This node's Multipath configuration will then be replicated to the other by copying /etc/multipah.conf
Print the current Multipath configuration, by default it's disabled

Enable and start the Multipath service

Put in place a very minimal configuration (just to see multipath device), the reload multipath and list devices in this configuration


 Let's rename these devices by giving them the more comprehensible names that are in the Disk table above. For that we're adding the following to multipath.conf and reload paths.


 Node2 can now be configured by replicating the Node1 configuration.


To complete with this subsection, we've to create a single partition on the Multipath disks. This procedure is mostly completed on one Node, with the exception of kpartx commands which are used to make sure that the physical path for the created partition exist on the nodes. Below is the output for 01 disk , this must be done for the all the shared disks.



 II.5 Shared Storage / Oracle ASMLIB

 Now that we have our shared disks well configured under DM-Multipath, we can move forward with ASMLIB Configuration. Below is a small recap of what we're aiming to complete in this sub-section.


Physical Disk
ASMLIB Disk Name
ASM Disk Group
ASM Redundancy
/dev/mapper/asm-crs-disk1p1
OCR_DISK1
OCR
High redundancy
/dev/mapper/asm-crs-disk2p1
OCR_DISK2
OCR
High redundancy
/dev/mapper/asm-crs-disk3p1
OCR_DISK3
OCR
High redundancy
/dev/mapper/asm-data-disk1p1
DATA_DISK1
DATA
External redundancy

Before diving in technical matters, let's first describe what is Oracle ASMLIB.  It is an optional support library for the Automatic Storage Management feature of the Oracle Database and allows an Oracle Database using ASM more efficient and capable access to the disk groups it is using. 

The Oracle ASMLib kernel driver is now included in the Unbreakable Enterprise Kernel.  No driver package needs to be installed when using this kernel.  The oracleasm-support and oracleasmlib packages still need to be installed from ULN. While the driver and support tools are on the Oracle Linux installation media, the oracleasmlib RPM is not (this package allows Oracle to access the kernel driver).  
Oracleasm-support is available on the Installation Media and on the Public Yum repository, That isn't the case for oracleasmlib which is only available on ULN. Subscribers to ULN can use yum(8) or up2date(8) to download and install the package on their servers. Non-subscribers are free to use the similar package built for RHEL on their Oracle Linux machines. In this case, we're downloading the similar package built for RHEL (available here)
Let's complete the Installation of Oracleasm-support and oracleasmlib.







There are some configuration to apply after ASMLib Installation.In fact, ASMLib is unaware of the multipath configuration. It can use any of the paths, sdb, sdc, or multipatha, and it won't know the difference. The default configuration does not care which path it uses.
In its default configuration, ASMLib will choose the first path it finds. This is the first path as reported by Linux. Depending on your storage driver, it could be the multipath, or it could be one of the single paths. We want ASMLib to always use the multipath disk and that must be told via its configuration.
On RHEL like system, the main configuration file is /etc/sysconfig/oracleasm (part of oracleasm-support). The following 02 lines are added to this file, the first is instructing ASMlib to scan the multipath disks (starting with dm under /dev) and the second to ignore all sd drives.

 ** /etc/sysconfig/oracleasm **


Note that when scanning the devices, only the device names known by the kernel are scanned. With device-mapper, the kernel sees the devices as /dev/dm-XX.
The /dev/mapper/XXX names are created by udev for human readability.So, Any configuration of ORACLEASM_SCANORDER or ORACLEASM_SCANEXCLUDE must use the dm prefix.

 We can now complete the oracleasm configuration by running the oracleasm command with its initial configuration switch (-i) , Note that this initial configuration will re-write the /etc/sysconfig/oracleasm (but will keep SCANORDER and SCANEXCLUDE configuration done before). This must be performed on both nodes.



Load ASM Module manually (on both nodes),



Now we can create our ASM Disk, this must be done only on one node.



Scan and list the newly created disks on both nodes,



With that done, we are done with this Section and can start the Oracle Grid Infrastructure Software Installation...
III. Oracle Grid 11GR2 Software Installation:

 Let's start the grid Infrastructure Software Installation by running runInstaller (needs graphical environment) as Grid User as described in the screenshots below.








At this stage, we will need to add the other cluster node, identify Network Interfaces and Setup/Test SSH Connectivity.



Click on SSH Connectivity, fill OS Password and run a Test and then setup.






Testing Again,



Click on "identify Network Interface", and mark both private/public Interfaces





With that completed, we can move to the next step where the Cluster Registery Storage Type will be set to "Oracle Automatic Storage Management" and we'll specify SYSASM password.




Just a few comments about the following  section (Disk Group Selection)
For Disk Group Name— Enter a name, for example: OCR_VOTE.
For Redundancy— For your OCR and voting disks, select High if five ASM disks are available, select Normal if three ASM disks are available, or select External if one ASM disk is available and you're managing redundancy in an external disk (not recommended).

Also, If no candidate disks are displayed, click Change Discovery Path and enter ORCL:*or /dev/oracleasm/disks/*. Ensure that you have marked your Oracle ASM disks, for more informations see,"Using ASMLib to Mark the Shared Disks as Candidate Disks"
Note that for the step below,  I faced the bug identified here ( Metalink Note 1267569.1) "Device check for ASM" and choose to ignore (as recommended).






At this stage, we should go back to the Shell and run the 02 scripts requested as root. For every script, run it on the local node first, after successful completion we can run in parallel on other nodes.

IV. Installing Oracle 11g R2 Database (RDBMS) Software:

As Oracle User,  let's run runInstaller






Click on SSH Connectivity and Setup/Test the connectivity












Again, we've to go back to the shell as root and run the following on both nodes.
With both Grid and Database Software Installation completed, the next step will most probably be the creation of Data Disk Group and DB Creation...


References:
http://sethmiller.org/it/oracleasmlib-not-necessary/
http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_solutions/3336.how-to-deploy-oracle-11gr2-on-rhel6oracle-linux-6
http://uhesse.com/2010/10/06/voting-disk-and-ocr-in-11gr2-some-changes/
http://dnccfg.blogspot.com/2012/09/koracle11gr2racsetup.html
http://oracleinaction.com/haip/
Publié par à l'adresse 4 comments:
Libellés : , , , , , , , ,

2014/10/28

Oracle VM for Sparc Autosave / Practical case

Have ever faced a situation where you think you've lost LDOMs configuration (mostly after an unexpected downtime)? If that's the case, then you must know how much autosave feature on Oracle VM for SPARC is important. Indeed, the main aim of the autosave feature is to ensure that a copy of the current configuration is automatically saved on the control domain whenever the Logical Domains configuration is changed. More important, it occurs even when the new configuration is not explicitly saved on the SP.

Note that this doesn't replace the classic LDOMs configuration Backup which consists of saving the constraints information for each domain into an XML file (ldm list-constraints -x...), but may supplement it when what is needed is just to recover domain Configurations that weren't saved to the SP. 

Enough talk! Let's move forward with this practical case. We've a System with 05 configured LDOMs, and while building them, an unexpected Hardware Failure occurs (:-)). The main issue is that we haven't saved yet our LDOM Configuration on the SP. Meaning before the Outage, we had something similar to:



And after the outage, we've something like this (with maybe some other parameters not present for the primary domain...).


Let's move forward with the following 05 little steps to recover our LDOMs configuration.

1. As a matter of precaution, take a Backup of the actual configuration and another one of the /var/opt/SUNWldm

 The /var/opt/SUNWldm folder contains the Autosave directories.


2.lists the available autosave configurations

 At this stage, we should have our autosaved configuration shown as being newer than the current configuration.

3. Recover the Autosave Configuration


4. Perform a full power Cycle

5. Check the LDOM Configuration 
Publié par à l'adresse 5 comments:
Libellés : , , , ,

2014/09/16

HP ILO / RHEL7 Systemd Output to VSP console

During my last post, I described the redirection of Linux Output to Serial on an Upstart Distribution based (using RHEL6). The main aim of this post is to describe the same on Systemd Distribution based (RHEL7). Before I begin with the technical matters, I have to say that I've been quite impressed by how Systemd makes this configuration so easy (no pain at all! So cool!). This isn't to make a comparison between Systemd and Traditional Init or Upstart, but you can check these two previous posts to draw your own Conclusions:  VSP/Traditional Init, VSP/Upstart.
Let's now delve in the interesting matters. The whole procedure is just about setting Kernel Options and reboot, and if the reboot can't be performed right away, just start a systemd service.

1. Set the Kernel Options boot options: 

On RHEL7 with Grub2, add "console=ttyS1" to  GRUB_CMDLINE_LINUX in /etc/default/grub file (You might also remove rhgb quiet as rhgb is for RedHat Graphical Boot and quiet is meant to hide the majority of boot messages before rhgb starts)



Changes to /etc/default/grub require rebuilding the grub.cfg file. This file location's on BIOS based machine is /boot/grub2 and for UEFI based machine, it is /boot/efi/EFI/redhat/.

On BIOS Based Machine:


On UEFI Based Machine:


2. Reboot or start a serial-getty service on ttyS1

Now we can either reboot the System to have the Kernel Loaded with the new parameter during the reboot , or (especially if we can't afford a downtime :-)) run the following to have a getty service started right away on the ttyS1.

That's it! So simple! Go on the console, run vsp and there's a nice prompt...


Reference:
http://0pointer.de/blog/projects/serial-console.html
Publié par à l'adresse 9 comments:
Libellés : , , , ,

HP ILO / RHEL6 Upstart Output to VSP console

Following my post related to HP ILO VSP console redirection , I've received many comments related to the same Configuration on recent Linux Distribution. In this post, I'm willing to detail the same on Upstart Based Distribution (using RHEL6, but should be the same for others Upstart Based distribution ). In fact, the main configuration on old Distribution was mostly related to Init Daemon Configuration, with new distribution based on Upstart/Systemd, things are slightly different. 

 Note that I won't detail the configuration of the Virtual Serial Port on BIOS/UEFI as it had already been discussed in this previous post.
If you're interested in the same configuration for Systemd Based Distribution, it's described in this post.

1. Create an init configuration file for ttyS1



2. Check the init configuration and start running the agetty process

 Upstart leverages the Linux inotify API to make itself aware of any changes that can happen within its configuration directory, so the creation of the ttyS1 file above is enough to have the service available and listed when using initctl. The only thing we have to do is to start the process.

3. Test you have access to the System through vsp

4. Add Serial Port to securetty to allow login as root

 This is needed if we want root account to be able to log in through this serial console.

5. Configure the Grub GRUB config file

 Finally, we can configure the GRUB to have outputs of the boot process on the console, this is easily achieved by adding console=tty0 console=ttyS1,115200
Publié par à l'adresse 1 comment:
Libellés : , , , , , ,

2014/09/11

Oracle/Redhat Enterprise Linux 7 Kickstart Installation / without DHCP

RHEL7/OEL7 is out for few months now, with a lot of new features (Revamped Anaconda, Systemd...). But before really starting to enjoy all these nice features, let's perform some basic Automated Kickstart Installation. In this short post, I'm willing to describe just that type of Installation without relying on a DHCP Server (using Static Network Parameters). I'm also using an Installation tree and Kickstart file located on httpd servers and reachable from the System I'm installing. Enough talk! let's detailed this Kickstart Installation in the following 4 steps:

1. Make Installation Tree available on an httpd server:

We've RHEL7/OEL7 ISO Files and an httpd server (192.168.0.10) configured (with DocumentRoot being the classic /var/www/html). Mounting the ISO as loop device in the DocumentRoot is enough to have the Installation Tree available over httpd.



2. Create the Kickstart File and make it available on the httpd system:

For that, I used as template an anaconda-ks.cfg  from another installed node and created the following kickstart.



Make this file available on http (copied under httpd DocumentRoot) and test to make sure it's reachable (i.e http://192.168.0.10/olnode.ks )

3. Start the Kickstart Installation

 I'm using an UEFI system, so In order to provide the right Kickstart Parameters during the boot process, I'm selecting an installation option in the boot menu and then press either the E key ( For BIOS Systems, that'll be he Tab key). A prompt is displayed which enables to edit the boot options already defined and to add new options.  In this case, I'm adding the following: 

 inst.ks=http://192.168.0.10/olnode.ks ip=192.168.0.20::192.168.0.1:255.255.255.0:olnode:eno49:none

 inst.ks specifies the location of the kickstart file, and ip sets statics network parameters, it must be in this form: ip=ip::gateway:netmask:hostname:interface:none. Below are some screenshots taken to illustrate this process.
Note that parameters such as hostname and Interface could be empty






4. Enjoy the Automation:

 Once that's done, the last step is to Press Ctrl-x (or enter on BIOS System) and enjoy the automation...





References:
http://docs.oracle.com/cd/E52668_01/E54695/html/ol7-install-boot-options.html

Publié par à l'adresse No comments:
Libellés : , , , ,

2014/08/25

Oracle VM for X86 / RHEL PVM Kickstart Installation without DHCP

Just a quick and short post to share a tip related to installation of RHEL as ParaVirtual Machine using kickstart(or any distribution that can use kickstart) on Oracle VM 3 for X86 without having to rely on a DHCP for the Network Parameters. The only prerequisite that is needed is to have kickstart file and RHEL Installation Path available on the network and reachable from Oracle VM Manager Server.
Once that's done, we create our Virtual Server as usual, except that for Network Path, we're having to add more parameters. Indeed we can use the --args to add any parameters we're willing to pass to the kernel. So, we're having something like :

--args kernel_boot_paramaters http://install_server/rhel6_4

 For example, let's say that my kickstart file is located on an http server and accessible via http://install_server/my_kickstart.ks, I also need to set 192.169.0.10 as IP of eth1 interface in order to get access to this kickstart file's URL and the RHEL installation tree is http://install_server/rhel6_4. Then, i'll have the following:

 --args "ks=http://install_server/my_kickstart.ks ip=192.169.0.10 ksdevice=eth1 netmask=255.255.255.0 gateway=192.169.0.1" http://install_server/rhel6_4




Note that the --args section are in quote, this is needed to clearly draw the boundary between what is kernel arguments and the installation tree's URL. 
Publié par à l'adresse No comments:
Libellés : , , , , ,

2014/08/19

Ansible Hosts / Install alternate upgraded Python version

Following the highlighted error below which happened while trying to configure a RHEL4 node to be manageable by ansible (and that i rightly attributed to the old Python 2.3.4 which is the default version of Python for RHEL4), i've decided to install an alternate Python version on the same RHEL node without impacting the already existing environment. Below i'm sharing the 8 steps I followed to have that working.
Note that although i'm describing this Python Installation in  relation with Ansible Integration, Step 1 to Step 6 can be used for the installation of an alternate Python Environment on any other Linux Systems.

Ansible's Error:
[stivesso@ansible-server ~]$ ansible my_ansible_node -m ping  --ask-pass
SSH password:
my_ansible_node | FAILED >> {
    "failed": true,
    "msg": "\r\nSUDO-SUCCESS-josewbugtgyoijxrgkuxihoejbjsbiuq\r\n  File \"/home/esso_s/.ansible/tmp/ansible-tmp-1406122775.05-117998270117858/ping\", line 1177\r\n    clean_args = \" \".join(pipes.quote(arg) for arg in args)\r\n
     ^\r\nSyntaxError: invalid syntax\r\n", 
    "parsed": false
}


1. Install Prerequisites on the target host



2. Get an updated python package



3. Untar/unzip the package on the target node



4. configure with the alternate path as option (here, i'm planning to install the alternate environment in /opt/python2.7) and compile/install



5. Export Shared Library and Bin Library in PATH and LD_LIBRARY_PATH on User profile 



6. Create ld.so.conf configuration file and run ldconfig to have library loaded system wide



7. Add the following to /etc/ansible/hosts on the Ansible Control Machine for the target node

  

8. Test again the ping module from ansible-server...

Publié par à l'adresse No comments:
Libellés : , , ,
Subscribe to: Posts (Atom)