It's all connected

Somehow

Combining Guzzle With Selenium

While working on tests for an application I needed to reset some values through a POST request before running the test.

To do this, I used the excelent Guzzle library and imported my Selenium cookies so that Guzzle could be the same user as Selenium.

Here’s the relevant parts of the test:

(selenium_guzzle.php) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
<?php

class MyTest extends \PHPUnit_Framework_TestCase
{

    static $session;

    public function setUp() {

        $browser = 'firefox';
        $capabilities = null;
        $url = 'http://localhost:4444/wd/hub';

        $driver = new Selenium2Driver($browser, $capabilities, $url);
        self::$session  = new Session($driver);
        self::$session->start();
    }

    public function testPost() {

        $this->login(); // omitted
        $client = $this->getGuzzleClient();

        $response = $client->post("/some/rul");
        ...

    }


    protected function getGuzzleClient() {

        $client = new Client($this->baseUrl);

        $cookiePlugin = new  CookiePlugin(new ArrayCookieJar());
        $cookies = $this->getWebdriverSession()->getAllCookies();
        foreach($cookies as $cookie) {
            $cookiePlugin->getCookieJar()->add(new Cookie($cookie));
        }
        $client->addSubscriber($cookiePlugin);
        return $client;
    }
    /**
     * @return WebDriverSession
     */
    public function getWebdriverSession()
    {
        return $this->session->getDriver()->getWebDriverSession();
    }


}

Testing Symfony2 Commands

Here is a simple sample test for a standard Symfony2 Console command:

use Symfony\Component\Console\Application;
use Symfony\Component\Console\Input\ArrayInput;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\NullOutput;
use Symfony\Component\Console\Tester\CommandTester;
use Symfony\Component\DependencyInjection\Container;

class RunSyncCommandTest extends \PHPUnit_Framework_TestCase
{

    public function testRunCommand() {
        // here you setup any mocks, services etc you'll need + define a container
        $container = new Container();
        $container->set('my_service', new SomeService());

        $command = new RunSyncCommand();
        $command->setContainer($container);

        // this one is important by adding an Application, you get the default
        // arguments
        $application = new Application("test", "1.2.3");
        $command->setApplication($application);

        // commandTester handles input and output
        $tester = new CommandTester($command);

        // note the command => 'f' part, easy to loose.
        $tester->execute(array('command' => 'f'));

        // do your asserts on the output here...

    }

}

Nothing special, the bits that are noteworthy are:

  • Create an Application object to get the default options (verbosity etc)
  • Use the CommandTester instead of calling $command->run() yourself.

PHPUnit Speed Tips

When your testsuite starts to grow you’ll soon find yourself wishing you had spent more time creating simple unittests with mockups and created less of those fancy Symfony2 functional tests.

Here are some tips that you can use to speed up any PHPUnit testssuite - with a Symfony2 bonus tip.

I start off with this Xdebug configuration:

zend_extension=/usr/lib/php5/20090626/xdebug.so

xdebug.show_mem_delta = 1
xdebug.profiler_enable_trigger=1
xdebug.max_nesting_level=600
xdebug.profiler_enable=0
xdebug.coverage_enable=0

Now, my research found tips on how to disable xdebug by adding and removing a config file from PHPs INI_SCAN_DIR, but I found that that was way too much hassle for me. Instead I enable xdebug by default and disable all options.( http://www.boxuk.com/blog/fast-phpunit-and-xdebug-code-coverage/)

Tip 1: Disable as much as possible of Xdebug.

For your normal tests, disable xdebug as much as you can:

<php>
    <ini name="xdebug.default_enable" value="0" />
    <ini name="xdebug.remote_autostart" value="0" />
    <ini name="xdebug.remote_enable" value="0" />
    <ini name="xdebug.overload_var_dump" value="0" />
    <ini name="xdebug.show_mem_delta" value="0" />
</php>

Tip 2: Only enable coverage when you need it.

Keep a separate phpunit file for coverage tests where you enable coverage:

<php>
    <ini name="xdebug.default_enable" value="1" />
    <ini name="xdebug.enable_coverage" value="1" />
    <ini name="xdebug.remote_autostart" value="0" />
    <ini name="xdebug.remote_enable" value="0" />
    <ini name="xdebug.overload_var_dump" value="0" />
    <ini name="xdebug.show_mem_delta" value="0" />
</php>

Bonus tip:

Originally this tip comes from Kris Wallsmith.

Add this to your Symfony2 kernel:

class AppKernel extends Kernel
{
    // ...

    protected function initializeContainer()
    {
        static $first = true;

        if ('test' !== $this->getEnvironment()) {
            parent::initializeContainer();
            return;
        }

        $debug = $this->debug;

        if (!$first) {
            // disable debug mode on all but the first initialization
            $this->debug = false;
        }

        // will not work with --process-isolation
        $first = false;

        try {
            parent::initializeContainer();
        } catch (\Exception $e) {
            $this->debug = $debug;
            throw $e;
        }

        $this->debug = $debug;
    }
}

And your tests will run much faster because Symfony2 doesn’t check if any resources have been changed between each time it starts a new kernel.

Patterns for Testing Asynchronous Javascripts.

Many javascript test frameworks have a consept of async tests where your tests follow this pattern:

someTestCode.js
1
2
3
4
5
6
    "test http request": function(done) {
        $.post("/some/url", {}, function(result, success) {
            asset.equals("ok", result);
            done();
        });
    }

I spent quite some time debugging a testcase looking like this one, can you spot the error?

The problem with the code above is that done() only gets executed on success. If your request fails, then your test will timeout instead of telling you what happened.

The fixed testcase looks like this:

someTestCode.js
1
2
3
4
5
6
7
8
9
10
11
    "test http request": function(done) {
        $.post("/some/url", {}, function(result, success) {
            asset.equals("ok", result);
            done();
        },
        function(error) {
            assert(false, "Ajax failed");
            done();
        }
        );
    }

At least now you test will fail in a way where you will understand what happened.

Lessons for writing JS modules.

Most JS code you want to write is bigger than this example. It is also usually hidden within a module. This implies that you should take note of the done function early when developing the module and make sure it is present and also that it is injectable. Here’s one way:

kraken.js
1
2
3
4
5
6
7
8
9
10
11
12
13
    window.kraken = function(done) {
        var done = done;

        return {
            fetch: function(url) {
                $.post(url, this.success, this.error);
            },
            success: function(arg, arg2) {
                done();
            }
            ...
        };
    }

Ok, but what if your interactions contain multiple async callbacks?

Unittest purists will tell you that you are testing too much, which is true, but sometimes you have a complex setup with more than one callback firing that you want to test. For this I use a countdownLatch:

someTestCode.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
function countdownLatch(num, callback) {
    return {
        num:num,
        callback:callback,
        done: function(msg) {
            this.num--;
            console.log("Done",this.num,  msg);
            if (this.num == 0) {;
                callback();
            }
        }
    }
}
    "test http request": function(done) {
        var latch = countdownLatch(2,function() {
                assert.equals(2, $("#sum").val());
                done();
                });
        var obj = window.kraken(latch);
        obj.fetch("1");
        obj.fetch("2");
    }

The latch above also lets me pass a messate to the done call so I can log progress, something that can be usefull for debugging.

Summary

  1. When you write new JS modules, add support for a done callback for testing.
  2. Remember to wire up errorhandlers for all requests.
  3. You can use the done function for debugging as well.

Injecting HTML With BusterJS

I’ve been working with the very promising new JS testframework named buster.js which seems to be shaping up to become the next great thing in JS testing.

One of the reasons for this I thing is because they try to give you the whole stack, not just a toy for testing your latest implementation of the fibonaci series.

Loading HTML into the DOM of the test is still missing and someting that is quite usefull. JS code does not live in a vacum. They will fix it, see this issue,but until that happens - here’s how you can do it.

First setup your config file:

buster.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
    var config = module.exports;
    var fs = require("fs");

    config["someTest"] = {
        rootPath: ".",
        environment: "browser", // or "node"
        libs:  [
         "src/js/mootools-core.js",
        ],
        src: [
            "src/js/my.class.js",
        ],
        tests: [
            "tests/js/test.my.class.js",
        ],
        resources: [
            {path: "/menu", content: fs.readFileSync('src/tests/fixtures/menu.html')},
        ]
    };

In this example I use Mootools, but that is because I’m using mootools on the project I’m working on right now. Pick your poison.

In the test file I load the html fragment and inject it into the dom.

buster.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
    buster.testCase("Test menu handling", {
        setUp: function () {
            var r = new Request.HTML({
                async: false,
                url: buster.env.contextPath + "/menu",
                onSuccess: function(responseTree, responseElements, responseHTML, responseJavaScript) {
                    var body = new Element(document.body);
                    body.adopt(responseElements);
                }
                });
            r.get();
        },
        "Test something with dom": function () {

        }
    }

Note: With this method, the menu.html file should not be a complete html page, but just an html fragment that you can load into the dom.

Howto Export From Mysql to Hadoop Using Sqoop

This is a quick howto for working with Apache Sqoop.

I used Clouderas packages of Hadoop and Sqoop for this work. Both the maven packages and the debs.

A basic Sqoop import - MR workflow consist of:

1. Dumping the table info HDFS as sequencefiles:

sqoop import --options-file ./sqoop-options.txt  --as-sequencefile --columns contentobject_id,item --table someTable  --class-name=com.example.MyClass --target-dir=/user/tarjei/db-sqoop/

2. Generate the class that is used in the sequecefile:

sqoop codegen  --options-file ./sqoop-options.txt --outdir src/main/java/ --class-name com.example.MyClass --table someTable

Create a Mapper that works with the data:

The mapper will have the format, LongWritable,Myclass:

SomeMapper.java
1
2
3
4
5
6
    public class SomeMapper extends Mapper<LongWritable,MyClass,Text,Text>{
        public void map(LongWritable id, MyClass, Context context) throws IOException, InterruptedException {
            String someValue = context.getValue();
            ...
        }
    }

Thats it. The parts I missed in the documentation were the part that Sqoop generates a class for you to use when reading the dump files.

Links

The Sqoop user guide is very usefull. Also the Apache Sqoop page.

Handy Symfony2 Aliases

While working with Symfony2 I’ve ended up defining a few aliases that might be of use for you. They make it a bit faster to do the things you do all the time, i.e. test and clear caches.

Here they are:

.aliases
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
PROJECT_DIR=`pwd`
alias cc="$PROJECT_DIR/app/console cache:clear"
alias ccp="$PROJECT_DIR/app/console cache:clear --env=prod"
alias cct="$PROJECT_DIR/app/console cache:clear --env=test"
alias cca="cc --env=prod ; cc --env=test"
alias itest="phpunit -c /$PROJECT_DIRapp/phpunit.xml.ci"
alias con="$PROJECT_DIR/app/console"
alias routes="$PROJECT_DIR/app/console router:debug"
alias log='tail -n 100 $PROJECT_DIR/app/logs/dev.log'
alias plog='tail -n 100 $PROJECT_DIR/app/logs/prod.log'
alias tlog='tail -n 100 $PROJECT_DIR/app/logs/test.log'
alias clear_prod_cache='frigg sudo -u www-data /webroot/vhosts/beta.scanmine.com/nywly/app/console cache:clear --env=demo'
#alias test="phpunit -c $PROJECT_DIR/app/phpunit.xml"
function test {
    
if [ $# -gt 0 ]
then
    echo "Filtering on: $1"
    phpunit -c $PROJECT_DIR/app/phpunit.xml --debug --filter $1
else
phpunit -c $PROJECT_DIR/app/phpunit.xml --verbose 
fi
}
# setup project perms:
# sudo setfacl -R -m u:www-data:rwx -m u:tarjei:rwx app/cache app/logs

Just a quick rundown:

  • con is an alias to the app/console object.
  • cc clears the cache
  • cca clears all the caches
  • routes lis an alias to the console router:debug command.

test is the command I use the most

It’s implemented as a function and the usage is as follows:

Run all tests

% test

Run the tests that hit by a filter

% test MyController::testIndex

Just remember that the aliases override the test function in bash so don’t reuse the same shell!

Converted to Octopress

I have converted this blog from using Drupal to using Octopress - I didn’t need a CMS to handle it. Being a developer, it is easier to handle everything using git and vim.

When I get the time, I’ll try to post the conversion scripts. A problem now is that a lot of codelistings do not look good at all. I hope to fix them soon.

Using Pear Packages With Symfony2

UPDATE: I improved the metod. See using Symfony2 with PEAR using Pyrus

I’ve just started using Symfony2 (http://www.symfony.com) and one of the first problems I stumbled across was how to integrate Pear packages into it.

The way I ended up doing this was to create a php directory in my verdors directory and install the packages into this directory using pyrus.

php pyrus.phar mypear ~/SymfonyProject/vendors
php pyrus.phar install pear/HTTP_Request2-2.0.0RC1

Then I added the directories to app/autoload.php:

app/autoload.php
1
2
3
4
5
    $loader->registerPrefixes(array(
       'Net_'             => __DIR__.'/../vendor/php',
       'HTTP_'             => __DIR__.'/../vendor/php',
    ));
    

This isn’t perfect as it clutters up the vendors directory with a lot of unneeded directories that follow a Pear installation - it would have been nicer if I could just download a cuple of phar files - so it is probably a good idea to have a separare pear directory under vendors to handle this.

Included file 'after_footer.html' not found in _includes directory