Add @Timestamp to your Python Elasticsearch DSL Model

The Python Elasticsearch Domain Specific Language (DSL) lets you create models via Python objects.

Take a look at the model Elastic creates in their persistence example.

 

#!/usr/bin/env python
# persist.py
from datetime import datetime
from elasticsearch_dsl import DocType, Date, Integer, Keyword, Text
from elasticsearch_dsl.connections import connections

class Article(DocType):
    title = Text(analyzer='snowball', fields={'raw': Keyword()})
    body = Text(analyzer='snowball')
    tags = Keyword()
    published_from = Date()
    lines = Integer()

    class Meta:
        index = 'blog'

    def save(self, ** kwargs):
        self.lines = len(self.body.split())
        return super(Article, self).save(** kwargs)

    def is_published(self):
        return datetime.now() < self.published_from

if __name__ == "__main__":
    connections.create_connection(hosts=['localhost'])
    # create the mappings in elasticsearch
    Article.init()

 

I wrapped their example in a script and named it persist.py.  To initiate the model, execute persist.py from the command line.

 

$ chmod +x persist.py
$ ./persist.py

 

We can take a look at these mappings via the _mapping API. In the model, Elastic names the index blog. Use blog, therefore, when you send the request to the API.

 

$ curl -XGET 'http://localhost:9200/blog/_mapping?pretty'

 

The save() method of the Article object generated the following automatic mapping (schema).

 

{
  "blog" : {
    "mappings" : {
      "article" : {
        "properties" : {
          "body" : {
            "type" : "text",
            "analyzer" : "snowball"
          },
          "lines" : {
            "type" : "integer"
          },
          "published_from" : {
            "type" : "date"
          },
          "tags" : {
            "type" : "keyword"
          },
          "title" : {
            "type" : "text",
            "fields" : {
              "raw" : {
                "type" : "keyword"
              }
            },
            "analyzer" : "snowball"
          }
        }
      }
    }
  }
}

 

That’s pretty neat! The DSL creates the mapping (schema) for you, with the right Types. Now that we have the model and mapping in place, use the Elastic provided example to create a document.

 

#!/usr/bin/env python

# create_doc.py
from datetime import datetime
from persist import Article
from elasticsearch_dsl.connections import connections

# Define a default Elasticsearch client
connections.create_connection(hosts=['localhost'])

# create and save and article
article = Article(meta={'id': 42}, title='Hello world!', tags=['test'])
article.body = ''' looong text '''
article.published_from = datetime.now()
article.save()

 

Again, I wrapped their code in a script.  Run the script.

 

$ chmod +x create_doc.py
$ ./create_doc.py

 

If you look at the mapping, you see the published_from field maps to a Date type. To see this in Kibana, go to Management –> Index Patterns as shown below.

 

 

Now type blog (the name of the index from the model) into the Index Name or Pattern box.

 

 

From here, you can select published_from as the time-field name.

 

 

If you go to Discover, you will see your blog post.

 

 

Logstash, however, uses @timestamp for the time-field name. It would be nice to use the standard name instead of a one-off, custom name. To use @timestamp, we must first update the model.

In persist.py (above), change the save stanza from…

 

def save(self, ** kwargs):
        self.lines = len(self.body.split())
        return super(Article, self).save(** kwargs)

 

to…

 

def save(self, ** kwargs):
        self.lines = len(self.body.split())
        self['@timestamp'] = datetime.now()
        return super(Article, self).save(** kwargs)

 

It took me a ton of trial and error to finally realize we need to update @timestamp as a dictionary key. I just shared the special sauce recipe with you, so, you’re welcome! Once you update the model, run create_doc.py (above) again.

 

$ ./create_doc.py

 

Then, go back to Kibana –> Management –> Index Patterns and delete the old blog pattern.

 

 

When you re-create the index pattern, you will now have a pull down for @timestamp.

 

 

Now go to discover and you will see the @timestamp field in your blog post.

 

 

You can go back to the _mapping API to see the new mapping for @timestamp.

 

$ curl -XGET 'http://localhost:9200/blog/_mapping?pretty'

 

This command returns the JSON encoded mapping.

 

{
  "blog" : {
    "mappings" : {
      "article" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "body" : {
            "type" : "text",
            "analyzer" : "snowball"
          },
          "lines" : {
            "type" : "integer"
          },
          "published_from" : {
            "type" : "date"
          },
          "tags" : {
            "type" : "keyword"
          },
          "title" : {
            "type" : "text",
            "fields" : {
              "raw" : {
                "type" : "keyword"
              }
            },
            "analyzer" : "snowball"
          }
        }
      }
    }
  }
}

 

Unfortunately, we still may have a problem. If you notice, @timestamp here is in the form of “April 1st 2017, 19:28:47.842.” If you’re sending a Document to an existing Logstash doc store, it most likely will have the default @timestamp format.

To accomodate the default @timestamp format (or any custom format), you can update the model’s save stanza with a string format time command.

 

def save(self, ** kwargs):
        self.lines = len(self.body.split())
        t = datetime.now()
        self['@timestamp'] = t.strftime('%Y-%m-%dT%H:%M:%S.%fZ')
        return super(Article, self).save(** kwargs)

 

You can see the change in Kibana as well (view the raw JSON).

 

 

That’s it!  The more you use the Python Elasticsearch DSL, the more you will love it.

Pass Bootstrap HTML attributes to Flask-WTForms

Flask-WTForms helps us create and use web forms with simple Python models. WTForms takes care of the tedious, boring and necessary security required when we want to use data submitted to our web app via a user on the Internet. WTForms makes data validation and Cross Sight Forgery Request (CSFR) avoidane a breeze. Out of the box, however, WTForms creates ugly forms with ugly validation. Flask-Bootstrap provides a professional layer of polish to our forms, with shading, highlights and pop ups.

Flask-Bootstrap also provides a “quick_form” method, which commands Jinja2 to render an entire web page based on our form model with one line of code.

In the real world, unfortunately, customers have strong opinions about their web pages, and may ask you to tweak the default appearance that “quick_form” generates. This blog post shows you how to do that.

In this blog post you will:

  • Deploy a web app with a working form, to include validation and polish
  • Tweak the appearance of the web page using a Flask-WTF macro
  • Tweak the appearance of the web page using a Flask-Bootstrap method

The Baseline App

The following code shows the baselined application, which uses “quick_form” to render the form’s web page. Keep in mind that this application doesn’t do anything, although you can easily extend it to persist data using an ORM (for example). I based the web app on the following Architecture:

 

 

The web app contains models.py (contains form model), take_quiz_template.html (renders the web page) and application.py (the web app that can route to functions based on URL and parse the form data).

[ec2-user@ip-192-168-10-134 ~]$ tree flask_bootstrap/
flask_bootstrap/
├── application.py
├── models.py
├── requirements.txt
└── templates
    └── take_quiz_template.html

1 directory, 4 files
[ec2-user@ip-192-168-10-134 ~]$ 

Install the three files into your directory. As seen in the tree picture above, be sure to create a directory named templates for take_quiz_template.html.

Create and activate your virtual environment and then install the required libraries.

[ec2-user@ip-192-168-10-134 ~]$ virtualenv flask_bootstrap/
New python executable in flask_bootstrap/bin/python2.7
Also creating executable in flask_bootstrap/bin/python
Installing setuptools, pip...done.
[ec2-user@ip-192-168-10-134 ~]$ . flask_bootstrap/bin/activate
(flask_bootstrap)[ec2-user@ip-192-168-10-134 ~]$ pip install -r flask_bootstrap/requirements.txt

  ...

Successfully installed Flask-0.11.1 Flask-Bootstrap-3.3.7.0 Flask-WTF-0.13.1 Jinja2-2.8 MarkupSafe-0.23 WTForms-2.1 Werkzeug-0.11.11 click-6.6 dominate-2.3.1 itsdangerous-0.24 visitor-0.1.3
(flask_bootstrap)[ec2-user@ip-192-168-10-134 ~]$ 

Start your flask application and then navigate to your IP address. Since this is just a dev application, you will need to access port 5000.

(flask_bootstrap)[ec2-user@ip-192-168-10-134 ~]$ cd flask_bootstrap/
(flask_bootstrap)[ec2-user@ip-192-168-10-134 flask_bootstrap]$ ./application.py 
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger pin code: 417-431-486

This application uses the quick_form method to generate a web page. Note that the application includes all sorts of goodies, such as CSFR avoidance, professional looking highlights and validation. Play around with the page to look at the different validation pop-ups and warnings.

Now imagine that your customer wants to change the look of the submit button, or add some default text. In this situation, the quick_form does not suffice.

Attempt 1: Use a Flask-WTF Macro

The Flask-WTF docs include a Macro named render_field which allows us to pass HTML attributes to Jinja2. We save this macro in a file named _formhelpers.html and stick it in the same templates folder as take_quiz_template.html.

{% macro render_field(field) %}
  <dt>{{ field.label }}
  <dd>{{ field(**kwargs)|safe }}
  {% if field.errors %}
    <ul class=errors>
    {% for error in field.errors %}
      <li>{{ error }}</li>
    {% endfor %}
    </ul>
  {% endif %}
  </dd>
{% endmacro %}

Now, update the take_quiz_template.html template to use the new macro. Note that we lose the quick_form shortcut and need to spell out each form field.

When you go to your web page you will see the default text we added to the field:

{{ render_field(form.essay_question, class='form-control', placeholder='Write down your thoughts here...') }}

And an orange submit button that spans the width of the page:

{{ render_field(form.submit, class='btn btn-warning btn-block') }}

You can see both of these changes on the web page:

Unfortunately, if you click submit without entering any text, you will notice that we have reverted to ugly validations.

Attempt 2: Use Flask-Bootstrap

Although pretty much hidden in the Flask-Bootstrap documents, it turns out you can add extra HTML elements directly to the template engine using form_field.

As before, we add default text with “placeholder:”

{{ wtf.form_field(form.essay_question, class='form-control', placeholder='Write down your thoughts here...') }}
{{ wtf.form_field(form.email_addr, class='form-control', placeholder='your@email.com') }}

We then customize the submit button. You can customize the button however you would like. Take a look here for more ideas.

{{ wtf.form_field(form.submit, class='btn btn-warning btn-block') }}

This gives us a bootstrap rendered page with pretty validation:

As you can see, we get a popup if we attempt to submit without entering text.

Conclusion

You now have a working web application that easily renders professional looking forms with validation and pop-ups. In the future you can trade ease of deployment against customability.