Fifthtry

Activity Store

Status: Under Development

Every HTTP request served by realm is an Activity.

Normally web frameworks write our a long entry about the HTTP request, which records http method, path, query parameter, IP address, user agent and so on.

These generic information is not very useful, and often requires parsing to make sense, and often doesn’t have information we need.

This project is about creating an activity framework.

Activity

struct Activity {
    url: String,
    referer: String,

    // instead of url, canonical id would be okind:ekind
    okind: String,
    oid: String,
    ekind: String,
    data: serde_json::Value,
    outcome: success | their fault | server fault,
    error: String,
 
    trace: serde_json::Value,
    trace_hash: String,
    response: ftd::Document,
    
    when: DateTime<Utc>,
    duration: Duration, // backend response generation time

    ip: String,
    user_id: Option<String>,
    session_id: Option<String>,
    visit_id: Option<String>,
    visit_created: bool,
    tracker_id: Option<String>,
    tracker_created: bool,

    app: String,
    site_version: DateTime<Utc>,
}

url is the URL that was accessed.

okind, oid pair decides the “object” on which the activity happened. In this framework all activities happen an object. If we do not have an appropriate object, say if the request represented the index page view, the okind can be “site”, and oid can be the URL (”/” in this case). For twitter, if /username/ page was viewed, the okind could be “user” and oid could be “username”.

ekind is the event that happened on the “object”. Eg user created, user logged in, user suspended and so on are the activities/events that can happen on the user object.

Some event’s or activities may have some associated data we may want to store, eg if user was suspended, it can contain “reason=SPAM”. This is stored in data.

success stores if the activity was successful, eg trying to login can either succeed or fail. In case of failure, the reason could be stored in data.

Realm comes with integrated tracing framework powered by observer. trace field stored the trace of the HTTP request. This is optional, can be done on sampling basis or all request or only for status code non 200, or for request taking more than say 10ms etc.

We can store the full response as JSON in response field. This may be triggered on and off depending on settings. Can be useful to see what user saw in a journey for example.

when represents the time when http request arrived.

ip is to store the remote ip address.

user_id and session_id are application defined opaque strings representing user id if user is logged in, and session id if user has a session.

app can be used to refer to remote client that was used to create the activity, eg if activity was done by browser, or by iOS or Android app.

site_version stores the deployed date of when the site. We could have used an opaque string here, but auto generating the string as part of deployment process is good to have feature, and easiest option is to use timestamp as it serves multiple reason. No site does deployment so often that two different deployment will share the same timestamp, so it is unique id too.

Frontend Activity

From the frontend an activity can be created using Realm.Activity.send : OKind -> OID -> EKind -> Data -> Cmd msg method. The frontend activities are fire-and-forget activities and do not have a confirmation.

In case device is offline, the activities are queued up (stored in browser local store), and send when internet comes online (when tab becomes active after activity has been queued and internet comes online).

Activity Meta Data API

Application developer can use the following to easily pass activity meta data to realm:

pub fn get(cin: &Cache) -> realm::Result {
    Page {
        some_data: 42
    }
    .with_activity("Welcome", "user", "42", "answered")
}

.with_activity() takes title (of the HTML page), okind, oid and ekind.

Realm will extract user_id and session_id from user data:

pub trait UserData: std::string::ToString + std::str::FromStr {
    fn user_id(&self) -> String;
    fn session_id(&self) -> String;
    fn has_perm(&self, perm: &str) -> Result<bool>;
}

Activity Data in realm::In

In many cases, especially for error handing it could be handy to first store activity data, other than success in In object, and at the end of request update the activity’s success status. This also allows us to incrementally store event data from different functions.

impl In {
    pub fn activity_ekind(&self, ekind: &str)
    pub fn activity_object(&self, okind: &str, did: &str)
    pub fn activity_data(
        &self, key: &str, value: serde_json::Value
    )
}

Activity Store

One developers have provided all the activity data, realm can store the activity data in configurable backend.

For now realm will use environment REALM_ACTIVITY_DB, and cargo feature flag activity_store to decide which database backend to use.

The activity data may go in a sync fashion (meaning before response is sent to client, which provides stronger durability), or async/fire and forget fashion (meaning less strong durability guarantee, but ensuring faster response time).

By default activities are logged on stdout instead of stored in DB. In case of logs, activity data query api and activity ui are not supported.

Activity Data Query Rust API

realm::Activity::find() returns a builder, which methods for filtering on any of the fields of activity, eg:

realm::Activity::find()
   .user_id(user_id)
   .session_id(session_id)
   .limit(10)?

.limit() finalises the query, and returns all the data matching the query.

Activity UI

Realm comes with activity UI at /realm/activity/ if activated. It is only accessible users who has UserData::has_perm("view_activity").

Table Of Content

What is Realm?

A Bit On Motivation

Routing is Hard

What does Realm do?

Backend Data And Type Safety

Tutorial

Quick Start Realm Tutorial

In Depth Tutorial (not ready)

Nix
Shell
Doit
Hello Rust
Hello Elm
Hello Static Files
Hello Server Side Rendering
Pre-Commit Hooks

Routing, Request And Response

Frontend, Data, Navigation, And APIs

How To Guides

File Upload

Backend: S3 File Upload
Authenticated File Serving
Frontend: Uploading Files From Elm

How to use storybook?

How to implement “loading..”?

Docs

Realm.In

Realm.Storybook.Story

realm::In

realm::Context

realm::Result

realm.magicSlice

realm::RequestConfig

Environment Variables

Internals - Only for Realm Developers, not Users

“Realm DATA”
iFrame Controller
Shutdown Routine
Testing Internals

Change Log

Get Realm Starter Working

Transparent Offline Feature

How to make http requests in Realm?

Development

Tutorial: ToDo App

Realm Testing

Enhance Realm Starter

Double Load Issue

Deploy To Heroku Button

End failure

Realm-Starter Github Template

Proposal: Tracker And Visit

Proposal: Activity Store

Proposal: Bundling

Proposal: Retry On Network Error

Storybook: Editable JSON

Storybook: Notes

Storybook: Reference

Backlog

Readings

Change Log

How to Publish

Testing

Code Snippets

Skip rustfmt For Some Section

Close Modal Dialog When Clicked Outside

Ignoring Lints In Python

Ignoring Lints (clippy and rustc warnings) In Rust

Handle DateTime in Rust & Elm

Handle CiText value read in Rust

Transport Enum Type to and fro Rust/Elm through JSON