wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

NoSQL schema design


A question that pops up frequently in developer discussions is "how to structure your data in a NoSQL way?". To shed a light on this, we have a look at the approach invented 50 years ago and still an all time favorite

Normalization

In a simple order example, we are looking at four tables:

  • Customer
  • Product
  • Order
  • OrderEntry

CLassic SQL schema

In this design, there are no duplicates and some simple SQL can list out all I need, for example the order value for a given order:

SELECT oi.order_id,
       SUM(oi.quantity * p.price) AS order_total
FROM OrderItem oi
JOIN Product p ON oi.product_id = p.id
GROUP BY oi.order_id
WHERE oi.order_id = 67111;

or the revenue per customer:

SELECT c.id AS customer_id,
       c.name AS customer_name,
       SUM(oi.quantity * p.price) AS revenue
FROM Customer c
JOIN Order o ON c.id = o.customer_id
JOIN OrderItem oi ON o.id = oi.order_id
JOIN Product p ON oi.product_id = p.id
GROUP BY c.id, c.name;

Read more

Posted by on 06 June 2024 | Comments (0) | categories: NoSQL WebDevelopment

Development Containers - the fine print


Development Containers are supposed to liberate your development environment from a specific local installation, like container technology liberated your runtimes (a.k.a YAMLed them into Docker or Kubernetes).

Helmchart Pun

Development != Runtime

Containerization for development has some overlap and quite some difference to containerization for production:

Topic Development Production
Mutability You alter container content Container is static
Network Use internal network Use internal network
Access Developer tools++ Browser / App
Containers multiple multiple
Volumes primary container binds projectdir
all others mount only
all: bind or mount
Configuration devcontainer.json, docker-compose.yml docker-compose.yml,Helm Chart
Scope Runtime & Tooling Runtime
Dockerfile Setup development environment Configure production

Insights

  • There are many getting started resources available: here, here, here, here and here. They are working examples, commonly strong on what and how, but light on why
  • There are plenty of templates to learn by example
  • There seem to be substantial differences how it works on different platforms, subtle and annoying
  • On macOS (14.4.1) with with the devcontainer plugin 0.364.0 mount binds would not work in auxiliary containers, only in the main app
  • I couldn't find any descrption which subset of docker-compose.yml is supported
  • The most interesting templates, for now, live on the Microsoft Artifact Registry, when you know how to look. Starting with those saves you tons of time and frustration
  • You will love code --list-extensions to figure out the extensions you have in your vscode (unless you are a code n00b and don't have any)

Read more

Posted by on 12 May 2024 | Comments (0) | categories: Container Docker Java NodeJS WebDevelopment

Maven build with multiple Java versions


Imagine, you are tasked with maintaining a Java application that needs to run on more than one Java version. You want to ensure that it compiles, tests and builds on all of them.
This is our story, buckle up, there are a few moving parts

The big picture

  • We use Apache Maven to drive the project using the pom.xml
  • The Maven Toolchains plugin controls the Java versions
  • Using <properties> ... </properties> and Build Profiles to adjust conditions for processing
  • Annotatiosn like @Only8 and @Only17 help to qualify tests
  • Our build tool (Jenkins or Github Actions) will use a container provided (in our case based on Redhat UBI)

Read more

Posted by on 16 April 2024 | Comments (0) | categories: Java WebDevelopment

nginx as ingress for Docker compose


In June I wrote about how to use Docker & nginx to deliver statically rendered brotli files for your web (frontend) application. It improves delivery quite a bid, but left me wonder: isn't there too much static WebServer involved?

Double hop to deliver static files

A typical web application using micro/mini/midi services looks like this:

A typical Docker configuration

It is common, easy and concerns quite separated. However it comes with a set of challenges:

  • nginx doesn't do http/2 on proxy_pass, so you miss the ability to serve static files directly with http/2
  • For static file we have two nginx involved
  • Each service needs to be exposed to the host at some port
  • The service architecture leaks to the host based nginx. SO any change in service needs an update to the docker-compose.yml AND the host based nginx configuration
  • the containers depend on that, external to them, configuration

So I tried to design a better way to handle this without going all K-in:

Docker configuration with ingress as component

This looed like a more promising approach:

  • Services could be addressed with their internal network name
  • Only Ports 80 and 443 of one container need exposure on the host
  • The nginx configuration inside the container is immutable and can't accidentially be reconfigured in production (your image comes from a pipeline isn't it)

Challenges

  • When trying to configure certbot, I initially tried using the --nginx parameter with an http challenge and shared data mounts. None of the attempts worked satisfactory, so at the end I settled on aDNS-01 using CloudFlare.
  • Since I wanted the nginx configuration to be inside the container image (and not on a data mount), a good understandig of nginx's configuration is necessary. The only persisted information was /etc/letsencrypt for the certificate and a secret for CloudFlare credentials
  • When the nginx configuration is statically configured for TLS, on initial load it will fail since the certs don't exist yet. Auntie Google suggested a manual run of certbot, but I favour docker compose up to handle everything
  • I ended up creating my own docker images, which was an epiphany: it absolutely makes sense to build a container image for single use instead of trying hard to make it configurable and vulnerable to mis-configuration

Read more

Posted by on 15 November 2023 | Comments (0) | categories: Docker nginx WebDevelopment

Quarkus and GraalVM starter


When Java is one of the languanges in your portfolio, you might have heard of Quarkus, an alternative to Spring Boot build on top of vert.x and GraalVM, a polyglot runtime for Java and other languages.

This article describes the getting started that worked for me.

Moving parts

The fist hurdle to overcome is to install all the bits and pieces. There are plenty of versions (Java 11 - Java 21) and methods (maven, CLI, packet managers), so it con be confusing what to pick. I tried most of them and created a mess and a deep appreciation for the "reset to snapshot" feature afforded by virtual machines. Instructions work for macOS, Linux or Linux on Windows.Here we go:

  1. install SdkMan.

The tool helps to keep your software development kits under control. From their website:

"Meet SDKMAN! – your reliable companion for effortlessly managing multiple Software Development Kits on Unix systems. Imagine having different versions of SDKs and needing a stress-free way to switch between them. SDKMAN! steps in with its easy-to-use Command Line Interface (CLI) and API."

curl -s "https://get.sdkman.io" | bash

You can thank me later. Side note: there are 16 different JDK offerings that can be installed, we are spoiled for choices

  1. install GraalVM

Currently, as of time of writing, there are three GraalVM distributions available. The Open Source, community supported GraalVM Community Edition, the commercial, Oracle supported Oracle GraalVM which requires a license in production and the ReedHat backed Mandrel. Mandrel is advertised as "specifically to support Quarkus". The Java 21 version was not yet available on sdkman, so I used the community edition. To stick with Mandrel I will use the container build option, later more on that

# List SDKs
sdk list java
# install GraalVM
sdk install java 21-graaalce
  1. Install Quarkus

This will install the Quarkus CLI and for good measure Apache Maven. The Quarkus CLI makes a pleasant developer experience

sdk install quarkus
sdk install mvn
  1. Install Docker

You can use Docker desktop (required a license for larger organisations) or Rancher Desktop (which also handles Kubernetes), Podman Desktop, any of the alternatives or the command line. New to Docker? There's plenty of fish

Now we are good to go. Skipping the Code with Quarkus tutorial lets build a n app in java and native put it into a container


Read more

Posted by on 13 October 2023 | Comments (0) | categories: Java WebDevelopment

Fun with Azure Active Directory & JWT


Active Directory has been the dominant standard for IT directories, even if it isn't the prettiest tree in the forrest. It's younger sibling ~~Azure Active Directory~~ Entra ID is a big player in cloud based Identity Providers (IdP). Unsurprisingly it behaves differently than the gold standard KeyCloak

JWT expectations

A Json Web Token (JWT) payload is a very losely definded JSON object with various claims. There is only a minimal consent of properties":

{
  "iss": "https://where-it-came-from",
  "audience": "https://where-it-should-be-valid",
  "iat": "DATE/TIME -> issued at",
  "exp": "DATE/TIME -> expiry",
  "scope": "space separated list of scopes",
  "email": "user's email"
}

The whole thing is (un)defined in RFC7519, sufficiently loose, so anyone can claim to be standard compliant and nothing is interoperable (just like ical). There is a list of known claims, but RFC7519 states: "None of the claims
defined below are intended to be mandatory to use or implement in all
cases, but rather they provide a starting point for a set of useful,
interoperable claims.
"

To ease validation of signatures, one can use an URL .../.well-known/openid-configuration which provides a number of needed properties:

  • various endpoint URLs for authentication and token exchange
  • issuer: The value corresponding to the iss property in a JWT
  • jwks_uri: URL to read the public key to validate signatures
  • scopes_supported: what scopes does the API support

Azure - same but different

When you setup Domino for JWT you need a series of specific conditions. The interesting parts from the documentation:

  • One of the JWT's "aud" (audience) claims must match the Domino Internet Site's host name
  • JWTs must contain a "iss" (issuer) claim matching the "issuer" returned from the OIDC provider's .well-known/openid-configuration endpoint
  • JWTs must contain a "scope" claim that includes "Domino.user.all"

When you follow KEEP's how to configure Azure AD you will find a set of pain points, in no specific order:

  • You can't remove claims you don't need
  • Azure AD will not issue a scope claim, but an scp claim
  • The aud claim is fixed to the "Application ID URI"
  • The iss claim in a token does not match the issuer property from well-known/openid-configuration
  • The jwks_uri URL does not return an alg property for the algorythm (nor did I find any way to request an Elliptic-curve signer)

So there's tons of fun to be had with Azure ~~Active Directory~~ Entra ID


Posted by on 29 August 2023 | Comments (4) | categories: JWT WebDevelopment

Primary Posture Applications


We use a multitude of applications per day. Each of them captures some level of attention and interaction. Alan Cooper coined the term Application posture, with the mainly used application being the sovereign application. I personally like the term primary posture application better and will use it in this post

Being primany

Since users spend most of their time in it, there's a willingness to become "senior intermediate experts". Shortcuts are learned, workflows get shared and a deeper understanding is desired. Depending on the nature of your work, very different application are your primary

  • for a graphic desiger it might be GIMP or Inkscape
  • a vlogger spends lot of time in OBS
  • The controllers spend their days in spreadsheets
  • The sales manager in CRM
  • Operations is fond of ERP
  • eMail and chat are strong contenders too
  • the Scrum master lives in Jira, while developers on the command-line and IDE

Primary posture by association

To cover anything else, aggregators were used. Trailblazer here was the Lotus Notes Client: One did everything in Notes, the main job and all the auxiliary and transient would be there. This consistency was attempted to recreate using portals and intranets (for inspiration what intranets can achieve, head over to The Nielsen Norman Group).

Auxiliary applications

You need to complete a task fast and want effortless results. An auxiliary posture helps with that. Adding an appointment in a calendar, booking a ride share, filing tax returns.

Auxiliary applications with a primary posture

One's primary application is another's auxiliary. This is a huge problem especially for bespoke applications. Typically they are comissioned by departments who will use them in "primary posture" (e.g. the leave management system gets commissioned by HR). The leave administrator will happily learn all bells and whistles, while mortal users are irritated by the complexity. I recall working on a leave management system where the initial application form had over 30 fields to cover all eventualities. We were able to convince the application owner to take a 2 form approach: the initial form had: coming, going, type of leave and optional "on behalf". 2 buttons were offered: "more" and "submit". "More" would lead to the 30+ fields form. We monitored usage for 6 month. Not a single time the larger form was submitted.

Multiple front-ends

To avoid the primary auxiliary trap, a clear API that separates UI from business logic helps. It allows to build smaller front-ends that are auxiliary in nature but don't compromize integrity. OpenAPI is your friend


Posted by on 21 August 2023 | Comments (0) | categories: Software

Passphrase Generator


Passphrases are considered easier to remember for humans and harder to crack for machines, famously explained in this comic:

Pasword strength

The challenge then is to have a good word list to pick from. There are various measurements on how many words one person would use which could be as low as a thousand. Note there is a huge difference between recognize and use.

Passphrases and dices

In a recent Toot exchange ospalh pointed me to Diceware, a method to use dice rolls and a word list to determine a passphrase. Usually one uses the regular 6 sides dices and 5 dices, which lets you pick from a 7776 member word list. The EFF published a version using the 20-sided dice from Dungeon and Dragons as well as various word lists.

Wordlists

An attacker who doesn't know that they are dealing with a passphrase, using conventional cracking methods stands little chance to decipher the phrase. However as the defender you must assume, they know your word list, so it is imperative to keep it long, while maintaining the odds to remember (in any case you can use some extra brain). SOme of the word lists you can find online:

Math.random() to replace dices

Let's roll (pun intended) our own passphrase generator. To make it a little more fun these are our constrains:

  • passphrase has 5 elements: 4 words and one 6 digit number
  • the number appears at a random position
  • elements are separated by a - (for readability, in active use you might just filter them out)

Read more

Posted by on 24 July 2023 | Comments (0) | categories: Java WebDevelopment

Keep your github container registry tidy


SO you drank the cool-aid, like me, and use GitHub Actions to build your projects and GitHub pacckages for your private containers, maven produced Jars, npm modules. Soon your honeymoon is over and you hit the storage limit of your account.

You need to clean up

Looking at the packages you will notice, that they are all there, all the version, in case of containers even the untagged ones. The root of the problem is equally the solution: a GitHub Action to delete package versions. The package is very flexible and well documented, outlining several scenarios how to put it to use

Things to watch out for

You have to decide when you want to put it to use:

  • on schedule, like every Friday
  • manual, pressing a button
  • on each build, when you add a new package

I also experienced that {{ secrets.GITHUB_TOKEN }} wouldn't work when the package you target is private, even when it is in the same repository. Once you know, it's not a big deal, just create a PAT and add it to the repository's secrets. You might want to add workflow_dispatch to all triggers, so you can test run them anytime.


Read more

Posted by on 18 July 2023 | Comments (0) | categories: Container Docker GitHub Java JavaScript NodeJS

Deploy private npm packages into private containers using github actions


GitHub Actions are rapidly becoming my favorite CI environment. Their marketplace has an action for everything. Sometimes it takes a little trial and error before things work smoothly. This is one of that stories.

Authentication is everything

Imagine the following scenario: you have developed a set of private TypeScript (or JavaScript) packages and have successfully deployed them to the private GitHub npm registry under the name @myfamousorg/coolpackage - where myfamousorg must match the repository owner (org or individual).

Now you want to use them in your application. That application shall be packed in a Container and made available in GitHub's private registry. All that automated using GitHub Actions.

You will need a PAT (or two)

In GitHub, head to the Personal access tokens / Tokens (classic) section of your developer settings in profile. You need to create tokens that allow you to handle packages.

GitHub Tokens

There are two places where you want to enter that token:

  • In https://github.com/[your-org]/[your-repo]/settings/secrets/actions create a key GIT_NPM_PACKAGES and copy your PAT there. You can pick any name, you will need it in the GitHub action later
  • In ~/.npmrc, your global settings for npm in your home directory. Don't put the info in the .npmrc in your git project.
prefix=/home/[your username]/.npm-packages
@myfamousorg:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=[here goes the token]

The prefix property allows you to run `npm install -g [package] without admin access.


Read more

Posted by on 16 July 2023 | Comments (1) | categories: GitHub JavaScript WebDevelopment