Author: Surid

How to Use Sticky Session for CSRF submission on Highly Scalable Cloud App in Haproxy

HINT: If you are a nginx fan and used it in mass scale, then, you must have done this using ip_hash (Nginx Documentation). It follows the same purpose for Haproxy. Difference and benefits of using Haproxy over Nginx for L7 proxy in a highly scalable and reliable cloud app would be a discussion for another day.

Case Discussion:

Suppose, you have a Cloud app, that is load balanced & scaled between multiple servers using Haproxy, for example:

101.101.101.101
202.202.202.202
303.303.303.303

Now, if your app has a submission form, for example, a poll submission from your users, then, there is an issue in this Haproxy setup.

Let’s say, an User A, requests for the app, and gets the data from the server 101.101.101.101, the CSRF token he gets for the poll submission to his browser, also maintains the app hosted on 101.101.101.101. But when he press the submit button, HAProxy puts him on 202.202.202.202 app, and the app hosted on 202.202.202.202 instantly rejects the token for the session as the session is not registered for that app. For such cases, we need to maintain a ‘Sticky’ session based on the cookie set by the right server. That means, if the cookie is set by 101.101.101.101, HAproxy should obey and give the user 101.101.101.101 until the cookie or the session is reset or regenerated.

How To Do That:

What we need to do, let haproxy write the server id in the cookie, and make the directive ‘server’ to follow the cookie. Please remember, there are couple of other way to achieve this. There is another way of doing this is called ‘IP Affinity’, where you make sticky session based on IP of the user. There is another based on PHP session value. Setting sticky session based on php session should also work. I preferred the cookie based sticky session, just on random selection.

So, to write the server id in the cookie, you need to add the following in the haproxy ‘backend’ directive as following:

backend app-main
balance roundrobin
cookie SERVERID insert indirect nocache

In the cookie directive, you can see, we are taking the HAProxy variable ‘SERVERID’ and inserting that to the cookie attribute. Now, all you need to do, is to configure your balancing IPs to follow the cookie, like the following:

backend app-main
balance roundrobin
cookie SERVERID insert indirect nocache
server nginx1 101.101.101.101 cookie S1
server nginx2 202.202.202.202 cookie S2
server nginx3 303.303.303.303 cookie S3

S1, S2, S3 are just 3 different names of the cookies for the specific servers. After the above is done, you can now restart and see Haproxy is following stickiness based on the session you have.

Just to find out, how to test if you are using laravel, try to regenerate the session based on the session() helper method as following:

session()->regenerate()->csrf_token();

You should be able to see the content loading from different web servers when the session regenerates. But it will persists when the regenerate session method is not called.

How to Skip WHM Initial Setup Wizard When Stuck After Upcp

If you have recently ran upcp and the WHM initial setup wizard is stuck in a URL like the following:

https://yourhostname.com:2087/cpsess*****/scripts/initial_setup_wizard1

And can not get away with it, here is the easy way to do it. Basically each setup wizard has a skip button and the button goes to initial_setup_wizard1_do, so only adding the _do at the of your initial_setup_wizard1 should do the job, like the following:

https://yourhostname.com:2087/cpsess*****/scripts/initial_setup_wizard1_do/

This should take you to the WHM home by letting you save some of the new WHM features and will not ask again for initial setups.

How to Do Full Page Caching in Laravel / How to Cache Views in Laravel

Most of the developers use Laravel Cache for database query result caching. Although, this is efficient, but the ultimate caching performance enhancement is achieved through FPC or Full Page Caching for web apps. Laravel doesn’t give any hint, neither describe how to do this in their documentation, which is why the article.

What is Full Page Caching?

Technically a full page caching means, to cache the html response from an app. In FPC, it is generally accepted to use the route/view as the cache key concatenating or mixing with the VERB in request header.

When a user requests for a route, we usually pull a controller behind the route to process and prepare several data before sending them to views for response. But what if the data hasn’t changed since the last request? That technically means the response hasn’t changed, right? This essentially says, you can cache the full response and skip the whole controller processing, even pulling the view, instead, only put the Cached data in the response. Theoretically, this is the best form of caching mechanism for ‘Web Based’ solutions like Ecommerce, Newspapers, Blogs etc. This technique is known as FPC or Full Page Caching.

Laravel Cache

Laravel is best known for it’s documentation. Although, the Laravel Cache documentation, only follows how to cache the database queries, not the views. To understand how to do FPC using Laravel, let’s first look at how our views are usually formed. 

class NewsController extends Controller {
    public function index() {
        $news = News::all();
        return view('news.index')->with('news', $news);
    }
}

Here the view() helper method, returns a Laravel View instance. It doesn’t return the html or renders one. So who does it? Laravel does it for you under the hood, and pass it to Response class. Now to cache the views, you have to return the html and save it to cache. There are basically two ways of doing it.

The easiest way is to use a function called ‘render()’ that is available to View class which returns the html of the created View instance. Here is how you may convert the above controller method to return from cache:

class NewsController extends Controller {
    public function index() {
        if ( Cache::has('news_index') ) {
            return Cache::get('news_index');
        } else {
            $news = News::all();
            $cachedData = view('news.index')->with('news', $news)->render();
            Cache::put('news_index', $cachedData);                                         
            return $cachedData;           
        }  
    }
}

This should be it, simple, ha!

Here is more! I looked at the laravel documentation a bit more, and I could find there is another way you can do the above. This is using the Response class. view method returns a Views instance, while Response instance is able to return rendered html based on view. Here is how to do this:

Response::view('news.index')->with('news', $news);

This also means our idea that Laravel does the rendering under the hood is a bit wrong, it basically shoots the views instance to a response instance (which it has to) and returns it, that put the rendered html in the final response. We can now cache the above output and serve for future requests without entering the controller’s processing!

How to Add a Zimbra User to Allow Distribution List Creation

NB: This is going to be another documentation purpose post.

A distribution list allow you to create a mailing list. So for example if you have a CRM with a member of 100, you want to add them to a list, so that you do not need to email each of them distinctively when required, instead, you keep a list with an email like [email protected] and allow somebody to shoot at that email, which would make sure all the CRM members get the email. In Zimbra, there is a feature called ‘Distribution List’. By default this is only permitted to admin user to create. But in case, you want to permit a user to create distribution list, you would need to use ‘zimprov’ command. Here is the reference:

# su - zimbra
# zmprov grantRight domain yourzimbradomain.com usr [email protected] createDistList

Fairly simple!

How To Renew & Deploy Let’s Encrypt SSL on Zimbra Server – 2020

Note: This does not seem to work on 2021. I have written another article on how to do this now: How to manually install/renew let’s encrypt ssl in Zimbra

Ok, there is a reason to put 2020 on the title. Because the process has changed since past. At this moment, I manage a Zimbra server with multiple domains in it, which won’t deploy the ‘other’ domains if not specified. The process is fairly simple, but I am keeping this as a documentation purpose, so that I don’t miss next time.

To renew the certificate for attached domains using certbot is fairly simple, just do:

# certbot renew

Once done, you you want to use the pre-hook and deploy-hook to do the patching and deploying as following using certbot_zimbra.sh

# certbot_zimbra.sh -p
# certbot_zimbra.sh -r -d 'your_domain'

Updated, certbot_zimbra doesn’t take this. ‘-n’ used to be taken as new and ‘-r’ for replacing, now, ‘-r’ is removed. Instead you can use ‘-e’ to specify new domains. So the command for replacement and deployment would be fairly simple as following:

# certbot_zimbra.sh -p
# certbot_zimbra.sh -d -e 'mail.yourdomain.com'
# certbot_zimbra.sh -d -e 'mailapp.yourdomain.com'

… and so on. At this moment, I couldn’t find a way to advise zimbra certbot to follow a list of domains instead of one. But this is probably possible by cracking the certbot.

Quick Tips: An error occurred. Your account may be over its quota or you attempted to upload a folder – Cpanel

An error occurred. Your account may be over its quota or you attempted to
upload a folder.

The error is very obvious. It means the account is over the quota. But what if it isn’t? This error is generic, cpanel throws this, whenever it fails to upload the file, regardless of what error it returns. There is a possibility that your IDS (Intrusion detection system) is discarding the upload, so double checking the IDS log should help you to conclude that. But what if, that is also not the case?

Ok, that can actually still happen. It happens when the customer uses cloudflare and uses cloudflare to login to the cpanel using cpanel proxy and then use it to upload the file. Cloudflare sees the upload going through web and blocks it. So, just double check the domain he uses to login to the cpanel, and check whether it uses some kind of 3rd party web application firewall loaded application or not like Cloudflare. If it does, that could be the case!

TDD: Date Assertions – Laravel 7, PHP 7.* – Carbon 2 – Changes

If you follow a TDD approach to develop your software, and also a Laravel user, working with a JSON API, you might have experienced some date assertion issues while asserting Json with Laravel 7. Previously, when Laravel was using Carbon 1 for date management, it would not return the whole Carbon object for assertJson. Which is why, the following would work in a sample PHPUnit Test:

$contact = factory(Contact::class)->create();
$response = $this->get('/api/contacts' . $contact->id);

$response->assertJson([
'name' => $contact->name,
'email' => $contact->email,
'meeting_date' => $contact->meeting_date,
'company' => $contact->company,
]);

But as of now, Carbon 2, returns the whole Carbon object for $contact->meeting_date here, the assertion will fair, because the $response, didn’t get a Carbon object, instead a Json string here.

If you go through the Carbon documentation here, you can see the following under the section ‘Migrate to Carbon 2’:

$date->jsonSerialize() and json_encode($date) no longer returns arrays but simple strings: "2017-06-27T13:14:15.000000Z". This allows to create dates from it easier in JavaScript. 

This is basically what you need to use if you are on Laravel 7 with Carbon 2. You need to serialize the data as Laravel is not doing it for you automatically here to fix this up. Just change the ‘meeting_date’ to the following:

'meeting_date' => $contact->meeting_date->jsonSerialize(),

and this should let your assertion pass.

Remember, you might not need to explicitly do this in your controller return until you are following a TDD based development where an assertion to pass is important to continue the process. Laravel resource would implicitly serialize your data before returning them from controller.

Dirty Odoo Hacks: How to Know If a Scheduled Action is Running

This blog post goes to my ‘Dirty Hack’ series, where I try to open the gross hacking attempts I practically use in several of my projects.

Odoo is my favorite piece of framework. Even though it has evolved as OpenERP, but the extend ability that Odoo gives, probably can take it anywhere, any other framework unable to, without massive change to their base architecture. This goes to the pros of Odoo, while the biggest cons of Odoo, is that it is not owned/developed by a major ‘English’ speaking company/people. If you go through the Odoo documentation, you can clearly understand the difference I am talking about. One key reason, Odoo is not often seen everywhere, probably because of poor documentation. Most of the things are not explained in details, as Odoo is a large piece of software, makes it difficult for developers to dig it down and extend.

Odoo has it’s own scheduling system, which they call ‘Scheduled Actions’. Odoo manages a model ‘ir.cron’ to manage the Scheduled Actions. If you look at the model details from (on debug mode)
Odoo >> Settings >> Technical >> Database Structure >> Model
you will see, there is no field it uses to detect if the scheduled action is running or stopped. It has a ‘state’ property though, which does not mean the status like many other odoo models. Here is a output of state property from Odoo shell:

>>> cron_id = self.env['ir.cron'].browse(18)
>>> cron_id.state
'code'

That says, it is telling you what kind of operation it is going to execute, here it is saying, this scheduled action is executing a ‘Python Code’ here.

There are times, when you might require to determine whether the scheduled action is running or not. In my case, it was to track down the some syncing issues, like to find out whether the syncing is doing it’s job properly through some custom methods. As Odoo doesn’t provide a way to do this, I had to find a dirty way to do it. Here is how I did it.

Odoo runs in Postgres SQL, which is basically an open source version of Oracle like Object Oriented Database system. Like many other OODS, Postgres also provide an operation called the following:

FOR UPDATE NOWAIT

From the oracle documentation, it means:

Oracle provides the FOR UPDATE NOWAIT clause in SQL syntax to allow the developer to lock a set of Oracle rows for the duration of a transaction.

When the scheduled action runs, it locks the specific row of the ir_cron table, of course it would because it has to update that specific row with nextcall and other field data. So if a cron is running, and you try to put a lock with another process, will basically fail, which means the cron is running, otherwise the opposite is true. Viola, that should work for us, isn’t it? Simple! Then again, you have to remember, the lock will put in place for the amount of time you specify, or you have to throw an early rollback (why not a commit? to be in safety to avoid any unwanted data being committed to the database during that session). Here is the simple method to do the whole process:

def _check_if_cron_running(self):
cron_id = self.env['ir.cron'].browse(18)
""" assuming the cron id is 18, you can do any kind of search, like searching by name of the scheduled action etc """
if cron_id:
try:
self._cr.execute("""SELECT id FROM "%s" WHERE id IN %%s FOR UPDATE NOWAIT""" % cron_id._table, [tuple(cron_id.ids)], log_exceptions=False)
""" self._cr.execute is used to execute a direct sql command, each ir.cron model data will have 'ids' to tell you which ids are selected, and _table to tell you what is the table name for the model, here it should return ir_cron """
self._cr.rollback() # we need to rollback and give the database cursor back to the other process ASAP
_logger.info("log and operate whatever here, this section is reached when the scheduled action is stopped or not running")
except psycopg2.OperationalError:
self._cr.rollback() # we need to rollback the errors and give control to the other process
_logger.info("log and operate whatever here, this section is reached when the scheduled action is running")

Remember, this is dirty, you are hitting direct SQL, on a near around 500 model based framework. So, please take care of your things before placing this in production. Rest assured, keep using Odoo for future! A brilliant piece of art is Odoo!

How to Empty a Model In Odoo / Mass Delete in Odoo / Mass Unlink

In Odoo, we use unlink() ORM method to delete a record. But if you are trying to empty a model or wants to delete multiple records, the best way to do it, is to use two steps.

First, search the records:

record_set = self.env['your.model'].search([])

Second, unlink them all at once instead of looping through them:

record_set.unlink()

If you have a list of ids, search them using the list:

ids = [1, 2, 3, 10, 11, 12]
record_set = self.env['your.model'].search([('id', 'in', ids)])

and unlink:

record_set.unlink()

Remember, there is no need to loop through this iterable object, odoo unlink does it for you.

TIPS: If you want to get all the ids in a model directly in a list, you can use the following:

record_list = self.env['your.model'].search([]).ids

How to Hide a Column Dynamically in a Parent Model in Tree View on Odoo

We all know how to hide a column in a model based on parent value matching in Odoo installation. I have already written a post on this here:

How to hide a column ‘dynamically’ in Tree View on Odoo

Above case works when you are trying a hide a column in a sale order line, or move line or invoice line. But what if, you want to hide a column in stock picking delivery tree view or the receipt view? The above method, will not work, because column_invisible only works when the parent is referenced. But in a tree view list like ‘Deliver Orders’ under Inventory Overview, if you want to hide a column based on condition, how do we do that?

The answer is, we use data from context and set the attribute ‘invisible’ to True based on the context value. We can either do that by using any existing context value or we can add a context value using a computed field property and match it in the xml.

I will discuss on how can we use the existing context key:value pair in hiding a column on stock.picking model.

First, let’s imagine, we have a character field on stock.picking called ‘woocommerce_id’.

class stock_picking(models.Model):
_inhert = 'stock.picking'

woocommerce_id = fields.Char(string="Woocommerce ID")

Now the xml for Inventory overview stock.picking trees would be inheriting stock.vpicktree

<record id="inherit_delivery_picking" model="ir.ui.view">
<field name="name">stock.picking.inherit.delivery.picking</field>
<field name="model">stock.picking</field>
<field name="inherit_id" ref="stock.vpicktree"/>
<field name="arch" type="xml">
<xpath expr="//field[@name='location_dest_id']" position="after">
<field name="woocommerce_id"/>
</xpath>
</field>
</record>

Now, the above will show woocommerce_id in all vpicktree view, like ‘Delivery Orders’, ‘Receipts’, ‘Internal Transfers’. But What if, you want to show this field to only Delivery Orders?

To do that, we need to know the value for ‘default_picking_type_id’. This value is automatically set on stock.picking model view in the context by Odoo (Hints: You may view all the keys available to context in a model, by creating a computed field, that writes self.env.context.keys() to a file or show it in the tree view to find the keys, and self.env.context.get(‘key_name’) to find the value for it).

If you check the different values set by that key for different pages, you can see, it is set to 1 for delivery orders, 3 for receipts and 5 for internal transfers. Now if you want to show the woocommerce_id field to only delivery receipts, we set the attribute invisible for that field when the context key value is not 1 like following:

<field name="woocommerce_id" invisible="context.get('default_picking_type_id') != 1"/>

Save it, upgrade the module, and see the magic! cheers!