Symfony 4 – REST API Unit Testing with Doctrine and Sqlite DB

There are plenty of benefits of using serverless architecture utilising 3rd party service. Lambda + API Gateway combo are a common one. I’d argue it is the way to go forward in most situations: https://martinfowler.com/articles/serverless.html.

Despite that, I think there are still few insights that we can gain by learning traditional web architecture for an API.


Architecture

The meat of REST API are basically just doing simple CRUD operations. Typically, we have few layers which are very similar between web frameworks:

  • Routing, which is where the list of routes are defined
  • Controller, which is where request validation is performed, data are retrieved from source database, data being processed and finally an HTTP response with some data is being returned

In bigger applications, we usually want split those huge controller responsibilities into a few more layers for long term maintainability:

  • Validation, which is where the validation of your request query parameters and payload content is performed
  • Repository, which is where queries or ORM calls being made to the source database
  • Data, which is where raw data being transformed to models for easier manipulation
  • Model, which is the representation of the data in the source database

Testing those layers separately can be tedious and are repetitive. I’d argue in most cases are unnecessary as most of them are to perform similar CRUD operations on every routes. In this post, I would like to show how we can perform end to end test of those layers, thus reducing the need of more granular testing strategies.

Config, Routes and Controllers

For simplicity in our example, we would stick all the logic in the Controller layer using Symfony 4 with routing annotation. Let say, we have a Product API that does few simple CRUD operations under MySQL and at the end we would like to perform and end to end test on all the routes.

config/packages/doctrine.yaml

doctrine:
    dbal:
        # configure these for your database server
        driver: 'pdo_mysql'
        server_version: '5.7'
        charset: utf8mb4
        default_table_options:
            charset: utf8mb4
            collate: utf8mb4_unicode_ci

        url: '%env(resolve:DATABASE_URL)%'
    orm:
        auto_generate_proxy_classes: true
        naming_strategy: doctrine.orm.naming_strategy.underscore
        auto_mapping: true
        mappings:
            App:
                is_bundle: false
                type: annotation
                dir: '%kernel.project_dir%/src/Entity'
                prefix: 'App\Entity'
                alias: App

src/Entity/Product.php

<?php

namespace App\Entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Table(name="products")
 * @ORM\Entity
 */
class Product
{
    /**
     * @ORM\Column(name="id", type="integer")
     * @ORM\Id
     * @ORM\GeneratedValue(strategy="IDENTITY")
     */
    private $id;

    /**
     * @ORM\Column(name="name", type="string")
     */
    private $name;

    /**
     * @return mixed
     */
    public function getId()
    {
        return $this->id;
    }

    /**
     * @return mixed
     */
    public function getName()
    {
        return $this->name;
    }

    /**
     * @param mixed $name
     */
    public function setName($name): void
    {
        $this->name = $name;
    }
}

src/Controller/ProductController.php

<?php

namespace App\Controller;

use App\Entity\Product;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;

/**
 * @Route("/products")
 */
class ProductController extends AbstractController
{
    /**
     * @Route("/", methods={"GET"})
     */
    public function listAction()
    {
        $products = $this->getDoctrine()->getRepository(Product::class)->findAll();
        $results = [];
        foreach ($products as $product) {
            /** @var Product $product */
            $results[] = [
                'id' => $product->getId(),
                'name' => $product->getName()
            ];
        }
        return $this->json($results);
    }

    /**
     * @Route("/{id}", methods={"GET"})
     */
    public function singleAction($id)
    {
        $product = $this->getDoctrine()->getRepository(Product::class)->find($id);
        if (!$product) {
            return $this->json([], Response::HTTP_NOT_FOUND);
        }
        return $this->json([
            'id' => $product->getId(),
            'name' => $product->getName()
        ]);
    }

    /**
     * @Route("/", methods={"POST"})
     */
    public function createAction(Request $request)
    {
        $data = json_decode($request->getContent(), true);

        $product = new Product();
        $product->setName($data['name']);

        $em = $this->getDoctrine()->getManager();
        $em->persist($product);
        $em->flush();

        return $this->json([
            'id' => $product->getId(),
            'name' => $product->getName()
        ], Response::HTTP_CREATED);
    }

    /**
     * @Route("/{id}", methods={"DELETE"})
     */
    public function deleteAction($id)
    {
        $product = $this->getDoctrine()->getRepository(Product::class)->find($id);

        $em = $this->getDoctrine()->getManager();
        $em->remove($product);
        $em->flush();

        return $this->json([], Response::HTTP_NO_CONTENT);
    }
}

End to End Testing

The essence of end to end unit testing is to perform these few basic steps on every single test case:

  1. Setup an empty data source environment
  2. Populate with initial data
  3. Perform HTTP request, ie. CREATE /products with payload data
  4. Validate the HTTP response code and JSON response returned are correct
  5. Validate the data source now has the correct data, ie. upon DELETE request, the actual table row is actually removed

Ideally, this would an independent unit test which we can run in our CI/CD instance and hence does not rely on external application (ie. database server) call to be performed.

This is where an in-memory SQLite + Doctrine combo comes in handy. Utilising an ORM means we are free from writing SQL specifics vendor, thus we basically can be confident that this testing approach would also work for our actual database vendor.

config/packages/test/doctrine.yaml

doctrine:
  dbal:
    driver: 'pdo_sqlite'
    url: 'sqlite:///%kernel.project_dir%/var/test.db3'
    memory:  true

src/DataFixtures/ProductFixtures.php

<?php

namespace App\DataFixtures;

use App\Entity\Product;
use Doctrine\Bundle\FixturesBundle\Fixture;
use Doctrine\Common\Persistence\ObjectManager;

class ProductFixtures extends Fixture
{
    public function load(ObjectManager $manager)
    {
        foreach (['BMW', 'Mercedes', 'Tesla'] as $name) {
            $product = new Product();
            $product->setName($name);
            $manager->persist($product);
        }
        $manager->flush();
    }
}

tests/AbstractControllerTest.php

<?php

namespace App\Tests;

use Doctrine\Common\DataFixtures\Executor\ORMExecutor;
use Doctrine\Common\DataFixtures\Loader;
use Doctrine\Common\DataFixtures\Purger\ORMPurger;
use Doctrine\ORM\EntityManager;
use Doctrine\ORM\Tools\SchemaTool;
use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
use Symfony\Component\BrowserKit\Client;

class AbstractControllerTest extends WebTestCase
{
    /** @var EntityManager $manager */
    private $manager;
    /** @var ORMExecutor $executor */
    private $executor;
    /** @var Client $client */
    protected $client;

    public function setUp()
    {
        $this->client = static::createClient();

        // Configure variables
        $this->manager = self::$kernel->getContainer()->get('doctrine.orm.entity_manager');
        $this->executor = new ORMExecutor($this->manager, new ORMPurger());

        // Run the schema update tool using our entity metadata
        $schemaTool = new SchemaTool($this->manager);
        $schemaTool->updateSchema($this->manager->getMetadataFactory()->getAllMetadata());
    }

    protected function loadFixture($fixture)
    {
        $loader = new Loader();
        $fixtures = is_array($fixture) ? $fixture : [$fixture];
        foreach ($fixtures as $item) {
            $loader->addFixture($item);
        }
        $this->executor->execute($loader->getFixtures());
    }

    public function tearDown()
    {
        (new SchemaTool($this->manager))->dropDatabase();
    }
}

tests/ProductControllerTest.php

<?php

namespace App\Tests;

use App\DataFixtures\ProductFixtures;
use App\Entity\Product;
use Doctrine\ORM\EntityManager;
use Symfony\Component\HttpFoundation\Response;

class ProductControllerTest extends AbstractControllerTest
{
    public function testListProducts()
    {
        $this->loadFixture(new ProductFixtures());
        $this->client->request('GET', '/products/');

        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_OK, $response->getStatusCode());
        $this->assertEquals($response->getContent(), json_encode([
            ['id' => 1, 'name' => 'BMW'],
            ['id' => 2, 'name' => 'Mercedes'],
            ['id' => 3, 'name' => 'Tesla'],
        ]));
    }

    public function testSingleProduct()
    {
        $this->loadFixture(new ProductFixtures());
        $this->client->request('GET', '/products/1');

        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_OK, $response->getStatusCode());
        $this->assertEquals($response->getContent(), json_encode(
            ['id' => 1, 'name' => 'BMW']
        ));
    }

    public function testSingleProductNotFound()
    {
        $this->client->request('GET', '/products/1');

        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_NOT_FOUND, $response->getStatusCode());
    }

    public function testCreateProduct()
    {
        $this->loadFixture(new ProductFixtures());
        $productName = 'Jaguar';
        $this->client->request('POST', '/products/', [], [], [], json_encode([
            'name' => $productName
        ]));
        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_CREATED, $response->getStatusCode());

        /** @var EntityManager $em */
        $em = self::$kernel->getContainer()->get('doctrine.orm.entity_manager');
        /** @var Product $product */
        $product = $em->getRepository(Product::class)->find(4);
        $this->assertEquals($productName, $product->getName());
    }

    public function testDeleteProduct()
    {
        $this->loadFixture(new ProductFixtures());
        $this->client->request('DELETE', '/products/1');

        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_NO_CONTENT, $response->getStatusCode());
        $this->assertEmpty($response->getContent());

        /** @var EntityManager $em */
        $em = self::$kernel->getContainer()->get('doctrine.orm.entity_manager');
        /** @var Product $product */
        $products = $em->getRepository(Product::class)->findAll();
        $this->assertCount(2, $products);
    }
}

Conclusion

Voila! There we have an end to end unit testing that we can run within our CI / CD platform and does not rely on a database server. This does not replace the need of full integration testing but it is significantly faster and easier to be run on regular basis and to test more complex logic. This is very important as it is considered good practice to adopt TDD approach in engineering teams.

Full source code can be viewed here: https://github.com/stellalie/api-unit-testing

Comments and feedback are appreciated 🙂

Notable Talks at LCA 2019

It’s the beginning of 2019 and it’s time of the year again for LCA 🙂 – Linux Conference Australia.

The original conference started to focus on discussion around Linux, however nowadays it also include talks on open source software movement and even just general engineering culture topics.

Me and my co-worker Andrew had the opportunity to go LCA 2018 at Sydney. At that time, I was contemplating to use my Learnosity training budget on LCA or those usual web development conferences, such Yow! or Web Directions. It was its the timing that made me ends up choosing LCA, it didn’t disappoint at all and I’m glad that I picked LCA as it offers more varied interesting talks and ideas outside web development topics.

This year, the LCA 2019 is held at Christchurch, NZ. Me and my co-worker Andrew got the opportunities to go again for the second time. The conference runs 5 days. I missed the first day as I was actually on leave and just got back from my flight overseas on the flight to NZ. We had our nights at a very nice AirBnb and had enjoyed our way throughout the week.

Interesting Talks

So these are my highly curated notable talks at LCA 2019 through the eye of a web developer that’s in my opinion is a must watch.

The Tragedy of systemd

The famous controversial talk by Benno. Of course.

Personal Branding for the Security Conscious

Design for Security

Web Security 2019

Database as a Filesystem

How to Disappear Completely

See the rest of talks: https://www.youtube.com/channel/UCciKHCG06rnq31toLTfAiyw

LCA 2020

Penguin Professional Dinner LCA 2019

I know. Baby steps… 🙂

I’m really looking forward for to be in Gold Coast next year! Watch this space: http://lca2020.linux.org.au/

Building a Game at Hack@LRN: Everyone Can Be a Hero!

The company I worked for, Learnosity, regularly held Hack@LRN as our take on a way for our product development team to take a break from everyday work, have fun, explore ideas our brilliant minds have pondered and experiment with new technologies.


We wanted to learn something new and have a bit of fun during the day. What would be a better idea, than building a game? So, we decided to put together a side-scroller, 2D shooter game, that relates to Learnosity.

We originally picked Unity as our game engine since it is a popular tool, and there are plenty of beginner tutorials. Due to licensing issues however, we weren’t allowed to use Unity Trial edition for our hackday. We ended up using our second choice, Godot, which is on an open source platform. We used our phones to take pictures then used Photoshop to create character faces, boss faces and background pictures. We used a free version of Spriter to create the character animations. For the rest of the assets (props, other monsters, musics, and sound effects), we relied on free versions we found on the internet.

We started our hackday by narrating a storyline, that went something like this. In the middle of our work day, while everyone was really busy developing apps, our office would come under an attack by aliens. Then one of us would come down along the streets of Wynyard, to fight monsters. Finally defeating the final boss at the Sydney Opera House.

Afterwards, we divided our tasks. Jack was the only one with any experience building a game before. So, he took on the task of wiring up the shooting mechanics, and the rest of us learned game development on that day by working on the level design, creating assets, and character animation. In the end, we all worked together. Using Godot to put our assets into the game and learned how to handle transition between scenes. Our biggest challenge was handling merge conflicts. We used git as our version control system, as we were all familiar with it, however it seemed that Godot didn’t play nice with it. Doing a git pull while Godot was still open, apparently, didn’t update the UI properly, so we ended up with everyone overriding each other’s work. We manage to get around this with good prompt communication to ensure everyone was working on the same version at all times.

Overall, it was a fun and satisfying experience! We learned a lot about game development in an extremely short period of time. 

Team: Jack, John, Michal, Stella
Tool: Godot, Photoshop, Spriter
Source: https://github.com/stellalie-co/godot-hackday

Screenshots

Learnosity office is under attack by invaders. We need your help!
Learnosity office is under attack by invaders. We need your help!
John is the only playable hero at the moment. Only so much you can do in a 12 hour hackday.
John is the only playable hero at the moment. Only so much you can do in a 12 hour hackday.
Our hero John must smash his way through, pawning all of the invaders around our office.
Our hero John must smash his way through, pawning all of the invaders around our office.
Our hero John is fighting the boss dragon Grace. Using lightning attack!
Our hero John is fighting the boss dragon Grace. Using lightning attack!
Our hero John climbs a building to get a better shooting position on the dragon boss Grace.
Our hero John climbs a building to get a better shooting position on the dragon boss Grace.
Our hero John climbs a building to get a better shooting position on the dragon boss Grace.
Our hero John climbs a building to get a better shooting position on the dragon boss Grace.

Gameplay Video

Recharge your Lightsail application generated by Bitnami with HTTPS and SSL

I started this blog by letting WordPress.com host it for $5 per month. I paid roughly $40 (with a coupon) for a year’s worth of hosting, got myself a free domain, great one-click install, and everything else works right out of the box. Great! or so I thought…

Then, I began to wonder, how can I remove the “Proudly powered by WordPress.com” shown below? Ugh, I can’t, it seems only business accounts, a.k.a $25 per month can do that. Ok, how about adding Google Analytics code? Ugh, you can’t inject anything into the head  tag either. What about promoting content via Adsense? Uhm, nope! Can’t inject that code either! Ok so how do I get around that? I figure, I needed to pay for a business account, so that’s $25 x 12 = a freaking $300 US dollars per year. That’s roughly 415 AUD per year down the drain. I don’t want to pay that much, at least not until I know my blog can generate some constant traffic, and is in need of their security updates, regular backups, optimisations, autoscaling, and other valuable features that these out of the box solutions offer. For now, this blog can work in a shared hosting environment and for that reason I don’t want to pay more than $5 bucks per month.

Here comes AWS Lightsail

Trying to get over my frustration with WordPress.com’s limited functionality, I began my search. I started to consider other options and stumbled on a $3.50 per month Lightsail application, with a WordPress one click install, powered by Bitnami. I gave it a quick spin, setup the domain names, and it’s seemed quite easy to do. Of course, it came with some DIY configurations that I had to do. I realised that it did not offer HTTPS right out of the box. uh oh… so, here’s some tips if you are like me and want to set these things up.

Why does HTTPS matter? It’s just a one-man blog

Besides the obvious security reason, it is also a usability and trust projection issue. Chrome will eventually label all websites without HTTPS as non-secure. You obviously don’t want your visitors to think that your site is not legit. Check out for more details: https://security.googleblog.com/2016/09/moving-towards-more-secure-web.html

In addition, HTTPS has been used in the Google search ranking algorithm since 2014, and I want my site to be indexed well.

Generating a Let’s Encrypt SSL certificate for your domain

On a serious note, the recommended way to use any serious Lightsail application, is to use the Lightsail load balancer, which costs $18 USD per month. This is basically why I went the hard way. $18Ă—12 = US $216. That’s exactly $298.75 AUD, and that’s way above my blogging budget.

https://lightsail.aws.amazon.com/ls/docs/en/articles/create-lightsail-load-balancer-and-attach-lightsail-instances

Let’s Encrypt is a Certificate Authority (CA) that issues free SSL certificates. You can use these SSL certificates to secure traffic to and from your Bitnami application host.

This guide walks you through the process of generating a Let’s Encrypt SSL certificate for your domain, installing, and configuring it to work with your Bitnami application stack.

Follow the very simple instructions here. Thanks to Bitnami for spoon feeding us with this tutorial:

https://docs.bitnami.com/aws/how-to/generate-install-lets-encrypt-ssl/

Redirect your HTTP to HTTPS

Now that you have installed your auto-renew certificate via cron, there’s one more step left to do. That is to redirect your HTTP traffic to HTTPS. 

This WordPress blog is a Lightsail application which is auto-generated by Bitnami. It’s running under Apache 2, so you will need to edit a prefix file and simply add htaccess rules below:

sudo vim /opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L]

If you are lazy like me, simply edit via the Lightsail browser terminal like so:

Then, restart your apps!

sudo /opt/bitnami/ctlscript.sh restart

Voila! Now all HTTP requests will be redirected to HTTPS, now my blog is uhmm… somewhat more secure, and ready for some SEO optimisation 🙂