Making HTTP POST Requests in PHP


Recently at my job I have had several situations in which I needed to pass fairly large amounts of data to a particular page on a website for processing, and in which submitting a form isn’t a feasible option. Typically in this situation one would attempt to append the data to the url using GET parameters, but sometimes your data is too large or too sensitive to pass through the URL. Unfortunately, PHP does not provide an easy way to make post requests natively. Luckily for us, there is cURL. cURL allows us to communicate across multiple protocols to a variety of different server types. With cURL, we are able to make HTTP POST requests:

function post_request($url, $data) {
	$output = array();
	foreach ($data as $key => $value) {
		if(is_object($value) || is_array($value)){
			$data[$key] = serialize($value);
		}
	}
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, $url);
	curl_setopt($ch, CURLOPT_HEADER, false);
	curl_setopt($ch, CURLOPT_POST, true);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
	$result = curl_exec($ch);
	if ($result) {
		$output['status'] = "ok";
		$output['content'] = $result;
	} else {
		$output['status'] = "failure";
		$output['error'] = curl_error($ch);
	}
	curl_close($ch);
	return $output;

}

The above function accepts two parameters, the URL to make the request to, and an associative array containing the keys and values you would like to submit with your POST request. The function returns an associative array containing two values: status, which will either be equal to “ok” or “failure” depending on the success or failure of the POST request; and either ‘content’ or ‘error’ depending on success or failure. In the case of a successful POST request, ‘content’ will contain the full output from the page you have posted to (In other words, you may need to be prepared for an entire html document). Otherwise, ‘error’ will contain the details of the error as returned by cURL.

Advertisements

Using Subversion to Automatically Publish Your Website


Recently at work my co-workers and I have been working on a fairly large and complex website (which will be top secret until early next year 🙂 ), and we have encountered one of the pitfalls associated with group web development: Overwriting each other’s code with FTP. The natural solution was to use Subversion to manage our source code. We created our repository, checked out our local copies, and regularly updated and committed our code. While this greatly reduced the frequency and severity of our FTP mishaps, it did not eliminate them entirely. As tension began to mount (exaggerated greatly for literary value), we decided to seek a better solution… something that would allow for our development server to automatically reflect the contents of our source repository. We eventually settled on the use of a post-commit hook, which for those of you who are new to subversion is simply a script (which can be written in any language your server understands) which is executed every time anyone commits to the repository. Unfortunately for us our Subversion Repository was hosted on a third party server, and we did not have access to the server to write the hook. This meant that we were going to need to create a new repository on a server we could access. We chose to use the Web Server we host our site on, just because it reduces the overall complexity of our setup considerably. The following steps were implemented on an Ubuntu 11.04×64 server, but the steps should be nearly identical (with the exception of possibly the package management) on any modern Linux server.

Using Subversion to Automatically Publish Your Website

Install Subversion and Subversion Apache Support

sudo apt-get install subversion libapache2-svn

Create the SVN Directories

You can place your SVN Root anywhere you would like on the filesystem, but I chose to place mine in /svn, feel free to edit these commands to reflect your desired location. Also, in my example I change the ownership of the svn directory to the user/group www-data. If you are using a different Linux distribution you may need to check to see which user Apache runs as by default, and adjust the commands accordingly.

sudo mkdir /svn
sudo chown www-data:www-data /svn

Creating Your Repository and Configuring Apache to Serve It

We need to configure Apache to know the location of our repository (which I realize we haven’t created yet), so point your favorite text editor (I’m a Nano guy) at /etc/apache2/mods-available/dav_svn.conf:

sudo nano /etc/apache2/mods-available/dav_svn.conf

By default almost everything in the file is commented out. While it is possible to set up your repository to allow anonymous reading (and is in fact the default option), I opted to require authentication for every action on this repository since this source code will most likely end up containing database user credentials. Since an example of setting up anonymous reading is in the configuration file, I will allow you to un-comment it if you would like to use it. Here is an example of the config file I am using:

<Location /svn/repository>
  # Uncomment this to enable the repository
  DAV svn
  # Set this to the path to your repository
  SVNPath /svn/repository

  # Authorization Data
  AuthType Basic
  AuthName "Subversion Repository"
  AuthUserFile /etc/apache2/dav_svn.passwd
  Require valid-user
</Location>

There are a few things to note about the above file. The first is the tag, which needs to point to the location we plan to create the repository in. Also note that SVNPath should point to the same location. The next section contains data about how to authorize users. I used basic .htaccess style authentication, with an auth file located at /etc/apache2/dav_svn.passwd. The last line before the closing tag states that only users with a valid username and password are able to access the repository.

Now that we have Apache pointed at the repository, let’s create it! In order to prevent permissions issues, we need to become the www-data user:

sudo su www-data

when asked for a password, enter your own password. Now that we are the correct user, let’s create the repository at the location we specified above.

svnadmin create /svn/repository
exit

Now that the repository exists and Apache is configured to see it, we need to restart Apache to refresh the configuration:

sudo /etc/init.d/apache2 restart

After a moment, try to point your browser to http://%5Byourdomain%5D.com/svn/repository. If everything is configured correctly you should see a password box pop up. Of course, there are no users authenticated for the repository, so at this point you can’t get in.

Creating Your SVN User Accounts

Now that the repository is set up, we need to have authenticated users to commit to and update from the repository. When creating users, the command to create the first user is slightly different from the command to create all other users. For the first user do:

sudo htpasswd -cm /etc/apache2/dav_svn.passwd username

and for the rest do:

sudo htpasswd -m /etc/apache2/dav_svn.passwd username

After creating a user for yourself, go back to http://%5Byourdomain%5D.com/svn/repository and enter your username and password. If all goes well, you should see a screen indicating that your repository is at revision 0.

Setting Up Your Website to Retrieve Data from the Repository

There are several way to do this next step, and several things to consider before choosing a path. If your website already exists, you may want to set up a new virtual host on your server to hold the new copy until you are satisfied with the performance of the repository, or if you have an up to date local copy you may feel free to simply delete your old directory on the server (as we will be remaking it in a moment). Another thing to consider will be subversion configuration directories inside your website. If your server is configured like mine, it denies access to .svn directories, and an svn checkout command should be an easy way to accomplish our task. Otherwise, you may want to consider using an svn export command instead. Since my server is configured to be safe with .svn directories, I decided to go with the checkout command. Since my website is located at /var/www/mysite, the commands to do a fresh svn checkout is:

sudo rm -rf /var/www/mysite
cd /var/www
sudo svn checkout file:///svn/repository mysite --username yourusername --password yourpassword

You should now have a basically empty directory, which has the distinct advantage of being able to be quickly updated to contain the contents of the repository.

Updating Your Website After SVN Commit

Now that the repository is set up and our website is pointing to a working copy, we can set up our post-commit hook to update the website every time a commit is made. Due to the somewhat complex nature of Linux permissions, we will be performing three main steps to accomplish this goal: Creating a script to update the working copy in our webroot, giving the Apache user (in my case www-data) the ability to run that script as root without a password, and then creating our hook to call that script as root.
First we create the script (Note: this command will open Nano to a blank file for you to edit and as soon as you save the file it will also make that file executable):

sudo nano /usr/local/bin/svn-post-commit-update-mysite && sudo chmod +x /usr/local/bin/svn-post-commit-update-mysite

And inside place:

#!/bin/bash
cd /var/www/mysite && sudo svn update

This code will simply navigate to the working copy we checked out in the last step, and run svn update. This should update the contents of that folder to reflect the contents of the repository.

Next we will give the Apache user the ability to run this script as root without a password by editing /etc/sudoers

sudo nano /etc/sudoers

and adding the following line:

www-data    ALL=(root) NOPASSWD: /usr/local/bin/svn-post-commit-update-mysite

Next we will create and edit the post-commit hook for the repository:

sudo nano /svn/repository/hooks/post-commit && sudo chmod +x /svn/repository/hooks/post-commit

And inside place:

#!/bin/bash
sudo /usr/local/bin/svn-post-commit-update-mysite 1>&2

Testing Your Setup

Now that everything is theoretically set up, all we have left to do is test. The best way to do this would be to do an svn checkout on your local machine, copy the local backup of your website into the folder you performed your checkout into, and then attempt to run a commit on your working copy. If all goes well, you should be able to point your web browser at your Website and see what you just committed.

Taking Control of Your PHP Sessions: Using Database Sessions


Introduction

Recently while working on a project for work, I encountered an interesting issue. I needed to develop a way to add and remove permissions from a User in a system we are building (which will be made open source as a service to the internet community once development is complete), but I needed the ability to push out these permission changes even if a user is currently logged in to the system.

As far as I know PHP does not provide a way to edit User sessions other than the $_SESSION array, which only applies to the current User’s session. After some research, I discovered that it is possible to change PHP’s default session storage method. This can be very useful if you are using a CDN to share server load between multiple servers while still allowing data persistence through the session. Another useful benefit of storing your sessions in the Database is the ability to alter User sessions (for any user) on the fly.

There are a few things to note about editing User Sessions on the fly:

  • If you aren’t careful you can inadvertently edit the wrong session, leading to security issues
  • The way session data is stored in the database is through a single serialized data field. This means that before editing a session variable you must either parse and change only the value you wish to change, or unserialize the entire session data string, edit the individual values, and then re-serialize the data for insert.
  • I have never tried the above method, and therefore cannot provide any code to accomplish this task. If this changes, I will edit this post with the relevant code
  • If you are only storing basic login type data in the session (which would be pulled from the Database), killing a User’s session and allowing them to log back in is the safest way to ensure that any changes to the session will populate correctly

Setting up the Database

The SQL in this tutorial is written to work with MySQL, so if you’re using something else just bear in mind that your syntax may need to be tweaked, especially in the DDL.

Create the Table:

CREATE TABLE SESSION (
id VARCHAR(32) NOT NULL COMMENT 'Stores the Session ID',
access INT(10) UNSIGNED NOT NULL,
data TEXT,
PRIMARY KEY (`id`)
) ENGINE=INNODB
ROW_FORMAT=DEFAULT;

Create a few PHP classes

This first class I built handles the Database connection, and is in two parts: Config.inc.php (for configuring the database variables) and MySQLDatabase.class.php:

Config.inc.php

<?php
define("conf_hostname", "localhost");
define("conf_username", "user");
define("conf_password", "password");
define("conf_schema", "database_to_use");
?>

MySQLDatabase.class.php

<?php
include("Config.inc.php");
class MySQLDatabase{
	private $db;
	private $hostname;
	private $username;
	private $password;
	private $schema;
	
	function __construct() {
		if(func_num_args() == 0){
			$this->hostname = conf_hostname;
			$this->username = conf_username;
			$this->password = conf_password;
			$this->schema = conf_schema;
		}
		else{
			$params = func_get_args();
			$this->hostname = $params[0];
			$this->username = $params[1];
			$this->password = $params[2];
			$this->schema = $params[3];
		}
	}
	
	private function open(){
		$this->db = mysql_connect($this->hostname, $this->username, $this->password) or die ('Error connecting to mysql');
		mysql_select_db($this->schema, $this->db);
	}
	
	public function executeQuery($query){
		$this->open();
		$results = mysql_query($query, $this->db) or die ("Error in query: $query. ".mysql_error());
		return $results;
	}
	
	public function close(){
		mysql_close($this->db);
	}
	
}
?>

Next we need to create a class to handle some session related events:

DatabaseSessionHandler.class.php

<?php
include_once ("MySQLDatabase.class.php");
/**
 *
 */
class DatabaseSessionHandler {
	private $db;

	public function _open($save_path, $session_name) {
		$this -> db = new MySQLDatabase();
		return true;
	}

	public function _close() {
		$this -> db -> close();
	}

	function _read($id) {

		$id = mysql_real_escape_string($id);

		$query = "SELECT data
				FROM SESSION
				WHERE id = '$id'";

		if ($result = $this -> db -> executeQuery($query)) {
			if (mysql_num_rows($result)) {
				$record = mysql_fetch_assoc($result);
				return $record['data'];
			}
		}

		return '';
	}

	function _write($id, $data) {
		$access = time();

		$id = mysql_real_escape_string($id);
		$access = mysql_real_escape_string($access);
		$data = mysql_real_escape_string($data);

		$query = "REPLACE
				INTO SESSION
				VALUES ('$id', '$access', '$data')";

		return $this -> db -> executeQuery($query);
	}

	function _destroy($id) {

		$id = mysql_real_escape_string($id);

		$query = "DELETE
				FROM SESSION
				WHERE id = '$id'";

		return $this -> db -> executeQuery($query);
	}

	function _clean($max) {
		$old = time() - $max;
		$old = mysql_real_escape_string($old);

		$query = "DELETE
				FROM SESSION
				WHERE access < '$old'";

		return $this -> db -> executeQuery($query);
	}
	
	public function killUserSession($username){
		$query = "delete from SESSION where data like('%userID|s:%\"". mysql_real_escape_string($username) ."%\";first_name|s:%')";
		$this->db->executeQuery($query);
	}

}
?>

Replacing PHP’s Session with our Database Session

Now all we have left to do is actually utilize our new session handling. This is actually a relatively painless process. In my application the session is started in the header include file for each page. In this case, all I had to do was change:

<?php
     session_start();
?>

to:

<?php
$sess = new DatabaseSessionHandler();
session_set_save_handler(array(&$sess, '_open'),
                         array(&$sess, '_close'),
                         array(&$sess, '_read'),
                         array(&$sess, '_write'),
                         array(&$sess, '_destroy'),
                         array(&$sess, '_clean'));
session_start();
?>

From here on out, the session can be handled in code exactly like a PHP session, but now you also have the ability to edit User sessions on the fly!

Learn to Program with Python pt. 1


Programming can be a fun hobby and a well paying career for those who learn to do it well, as well as a great way to keep stimulating your brain to keep you sharp and alert. Learning to program on your own can be difficult for some people however. Many programming tutorials have one of the following issues:

  • Talking above your head: Many programming tutorials assume a great deal of knowledge beforehand, and are therefore not suitable for brand-new programmers
  • Being half-heartedly written: Many tutorials start out well, but as the tutorial drags on the author loses interest, and the writing and content suffer dramatically
  • Generally sucking: That pretty much sums it up

I have decided to take my own crack at a programming tutorial, and I will attempt to avoid the previously mentioned pitfalls. There are a few details I would like to cover before we get started. The first is my choice of language. I have chosen to use Python for this tutorial for several reasons, the first and foremost being that I really enjoy programming in Python. I also think Python is great about getting out of your way and letting you program without having to micro-manage details such as Data Types (most of the time) and memory allocation (don’t worry if you don’t know what this means). The next detail I would like to cover is the timeline for this tutorial. I plan on writing this tutorial as multiple blog posts over the next several months, with many unrelated posts in between. If it has been a while, feel free to drop me a comment nagging me for the next issue.

Enough introduction, let’s get started with Programming and Python. To get started, we need to make sure we have Python installed on our system. If you are running Linux, chances are Python is pre-installed. For Windows and Mac users, I recommend ActivePython. ActivePython will install Python to your system, as well as a very basic Python editor (although any text editor will do). Open up your editor of choice and type the following:

print "Goodbye Cruel World"

Now save this as emoProgram.py . This is your first program. Kinda lame isn’t it? Don’t worry, you’ll be writing slightly less lame programs in no time. In fact, the only reason I had you do this is so I could teach you to run your program once you have written it. If you are using PythonWin editor (the editor that came with ActivePython) press Ctrl+R and a box will pop up asking you which file you want to run, and it should have emoProgram.py selected. Press OK, and it will run. If you have chosen to use a different editor (or even if you just like the command line), open up a command prompt and navigate to the folder where you saved your script and type:
python emoProgram.py
and you will see “Goodbye Cruel World” print to your terminal.

In an attempt to make each section as short as possible, I will be stopping here for now and picking up again in a few days with variables, operators, and data types.

My first “real” Python Program pt. 2


In the first post about my first real Python program, I talked about how I parsed out stories based upon a tag-based markup system, prompted the user for values for all of the tags, and then returned the story with all user input in place. Today I will be talking about how I created a system to allow users to write their own stories and have them automatically detected and parsed, as well as allowing users to save their created stories.
(more…)

My first “real” Python Program pt. 1


Whenever I start learning a new programming language, I like to start off with a project to get used to the syntax of the language without having to deal with finding holes in my logic. Typically, this program has been a simple “Mad-Libs” style program in which I prompt the user for some words and insert them into a (sometimes) hilarious story. For the most part in the past, these have been statically written stories in which I hard code variable names into a string and write custom prompts for each variable. When starting out in Python, I decided this would not be enough of a challenge. Somewhere between PHP and C# in my language path I hit a point where programming started to make sense on a deeper level. Adding new object types to my repertoire was becoming an everyday occurrence, I began branching out from what I had been taught in programming classes, and creating and dealing with objects dynamically in loops without ever knowing their names became the norm, and no longer blew my mind.
(more…)

Python “yield” and Project Euler problem 2


When working on Project Euler problem 2, I stumbled on a unique feature of Python: the yield keyword. The yield keyword allows for the creation of generators. Generators are defined much like functions, but have the added bonus of being able to return multiple values over the course of their execution. For example, lets say I wanted to make a function that counted from 0 to infinity, and kept it’s state between uses. The following generator would do just that:

def counter():
     a = 0
     while True:
          yield a
          a += 1

Simple isn’t it? Iterating through it is simple too. Let’s say we want to get the first 10 numbers, and then the next 20, with a break in between:

if __name__ == "__main__":
     x = counter()
     for i in x:
          if i < 10:
               print i
          else:
               break
     print "Broken out of counter"
     for i in x:
          if i < 30:
               print i
          else:
               break

The output would be:

0
1
2
3
4
5
6
7
8
9
Broken out of counter
11
12
13
.
.
.
23
24
25
26
27
28
29

Now to make this relevant. Project Euler problem 2 is as follows:

Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with 1 and 2, the first 10 terms will be:

1, 2, 3, 5, 8, 13, 21, 34, 55, 89, …

Find the sum of all the even-valued terms in the sequence which do not exceed four million.

So what we need to do is find a way to take the starting values of 0 and 1, and as each iteration passes, set a to the value of b, and b to the value of a+b, returning a each time. Sounds like a job for generators!

def fibonacci(max):
     a, b = 0, 1
     while a < max:
          yield a
          a, b = b, a+b

Next, we loop through each of the returned values, checking if it is even. If so, add it to a running total.

def sumEvenFibonacciNumbers(max):
     number = 0
     for n in fibonacci(max):
          if n%2 == 0:
               number += n
     return number

if __name__ == "__main__":
     print sumEvenFibonacciNumbers(4000000)

There you have it, using generators Project Euler problem 2 was a breeze!

Reading RSS feeds with Magpie RSS


Last semester I was taking a client-side programming class and ended up attempting to design a Twitter-esque web app using PHP and AJAX. The goal of the website was to create a twitter like interface for a news aggregation website. While my team and I were all competent programmers, there was one problem that we were all having a hard time solving at first: How to add realistic looking articles to our database while on a serious time crunch. After a little bit of debate and quite a bit of head-against-wall banging, we came upon the idea of pulling RSS feeds from major news organizations and parsing them out into byte-sized pieces (typically just the headline). We decided to use 3 major news organizations for simplicity’s sake since each RSS feed was just a little bit different, and we ended up going with NPR, CNN, and MSNBC World News.
(more…)

Getting Started with Project Euler


If you haven’t already, go ahead and sign up at Project Euler. Once you have logged in, head on over to the Problems page. I recommend sorting the problems by ascending difficulty, as this will allow you to tackle the problems in a (more or less) gentle slope, at least at first. Let’s take a look at the first problem:

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.

Find the sum of all the multiples of 3 or 5 below 1000.

So where do we get started? The key to this problem is the modulo operator. For those who are unfamiliar with the modulo operator (notated in many languages as % ), it divides two numbers and returns the remainder. So, if a number divides evenly by 3 or 5, we want to add it to a running total. So, if we were to look at this problem in language-agnostic pseudo-code we would get:

for each number between 0 and 1000
     if number/3 evenly, or number/5 evenly
          add the number to a running total
print the total to the user

With that, you should be able to complete this problem fairly quickly. If you would like to see this problem implemented in a few programming languages, hit the Read link below:
(more…)

Using SQLite with C#


Overview

Adding a database to your application can be an easy way to store data and settings between sessions for your program, but it is not always feasible to use a server based DBMS to store your database. SQLite is a small, fast, and reliable database which can be used without the end user having to install anything extra (achieved by referencing a single .dll in your project). There are a few things we as developers must do to get started with SQLite:

(more…)