Using Docker images as vehicles for binary deployment artifacts

I've seen it done before but wouldn't call it common. There are other artifact servers, and many git repositories have the concept of a release with artifacts that makes more sense.
I suspect Docker Hub is one of many artifact servers to introduce rate limits and other usage caps, so if you do this, make sure you control (or pay for) the registry server. Otherwise the risk is you'll exceed limits and lose access to your artifacts.
If you only want to copy binaries, then I'd skip the base image and use FROM scratch.
It's also possible to bypass the docker engine entirely and pull the artifacts from the registry server directly to the filesystem. I have a regclient project that ships a regctl command with regctl image manifest and regctl layer pull commands. E.g.:
$ regctl image manifest regclient/regctl
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 3019,
    "digest": "sha256:e3abac7c7ce78aaa5efddb5abedd54f8ed3077d9d5b03b5a66ecf9f4c3502454"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 930,
      "digest": "sha256:41437cd6a6e3d3098329474c62ccc5bd01fcf6b41bf35f0e8c89db96110b23ac"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 122413,
      "digest": "sha256:69d6afda22ccf4f223c016bfd3a108fe0a1ddd15c72978bd238b19201e0e1973"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 161,
      "digest": "sha256:57dd9736a25ab0b55d9d364dc20ee0946a844a8ac72d7f986ef902826f933e89"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 3963283,
      "digest": "sha256:282c5a3b4b6c8bf0b7038204919bbfc26c8b120cd87f4a1f573c12c808774c1c"
    }
  ]
}

$ regctl layer pull regclient/regctl sha256:282c5a3b4b6c8bf0b7038204919bbfc26c8b120cd87f4a1f573c12c808774c1c | tar -tzvf -
-rwxr-xr-x root/root  10235904 2020-11-05 20:52 regctl

Note that the sha's are unique and persistent, so if that hash is unchanged there's no need to pull the binary again. And the above example has a few extra layers since the command is expected to run in the image as a non-root user (/etc/passwd, ca-certificates, and a home directory). For a non-running image, that could be a single layer. E.g.:
# change image to your scratch-based source
image=regclient/regctl
store=/tmp
sha=$(regctl image manifest $image --format '{{ (index .Layers 0).Digest}}')
if [ ! -d "$store/$sha" ]; then
  mkdir -p "$store/$sha"
  regctl layer pull $image $sha | tar -xzvf - -C "$store/$sha"
fi
echo "result in $store/$sha"


What is the proper relationship between rollback/rollforward and MTTR metrics?

Yes, MTTR is/should always be tied to the business outcome: if things are not stable the very business is at risk.

The fact that the expected code/feature/change is still stuck in process in scenario 1 is irelevant: the feature is not stable, so it doesn't bring new business, rolling back is the best you can do at that time from the business prospective.

The rollforward is a gamble: keeps the business at risk waiting for a potential fix that in fact has statistically lower changes of success (due to the instability it will always be rushed compared to the change that caused the instability in the first place without even having such pressure on it). The rollforward is a yet another version of the code which hasn't been checked before.

If you want to keep the MTTR low you rollback immediately, without debate. This removes the business risk and gives you a chance to check that the fix is actually working before attempting to deploy it. I'd strongly suggest making it a policy as yes, almost always there will be someone asking for a fix instead of the rollback and calling a meeting to negociate/decide on it - all while business remains at risk.

Side note: if you're concerned with a high Change Failure Rate then I'd suggest checking the the actual rollback rate instead of deriving it from a low MTRR. Maybe you'd like to add a gate check before deployment for the most frequent failures.
If you have such check already automated - why not include it in the CI verification? If you don't have one - maybe its time to start thinking about it? :)


Pass environment variables from pipeline job A to pipeline job B

The issue was missing comma between build job and parameters. Below is the correct syntax:

...
                    build job: 'run-on-single-repo',
                    parameters:

...


Get the latest version of Tomcat:

$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=1' | jq -r .[].name
9.0.20

Get the latest version of Tomcat 9:

$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^9 | tail -n 1
9.0.20

Get the latest version of Tomcat 8.5:

$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^8.5 | tail -n 1
8.5.41

Get the latest version of Tomcat 7:

$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^7 | tail -n 1
7.0.94

These all currently match the latest versions listed at http://tomcat.apache.org/


Why is the K8s UI not available when rancher is used?

I've seen this happen, especially on the newest (as of now) version. Try going into the "Infrastructure stack" and look for stopped containers. Often it helps to just refresh the entire deployment (hit the button "up to date" next to the "kubernetes" stack and hit save (If I remember correctly) in the bottom to force the "refresh").

You could also try manually just restarting the kubernetes-dashboard containers.

Lastly you could use the kubectl cli to delete the dashboard pods and force them to get recreated.

Let me know how it goes! I know I had to do this fix a few times.


How to mark build success when one of the stages is aborted?

I answered that question stackoveflow and I'll post the answer here:

For a Declarative pipeline (not a scripted) you would do something like this

stage('Foo') {
      steps {
            script {
                env.PROCEED_TO_DEPLOY = 1
                try {
                    timeout(time: 2, unit: 'MINUTES') {
                        // ...
                    }
                } catch (err) {
                    env.PROCEED_TO_DEPLOY = 0
                }
            }
      }
  }
  stage('Bar') {
    when {
        expression {
            env.PROCEED_TO_DEPLOY == 1
        }
    }
    //rest of your pipeline

stage "Bar" would be skipped but as for the rest of the stages the job will be marked as passed (assuming nothing wrong happened before).

As for your particular use-case, the trick is to use a try/catch block.


Variables have different priorities, see vars precedence.

As of Ansible 2.x:

  role defaults
  inventory file or script group vars
  inventory group_vars/all
  playbook group_vars/all
  inventory group_vars/*
  playbook group_vars/*
  inventory file or script host vars
  inventory host_vars/*
  playbook host_vars/*
  host facts
  play vars
  play vars_prompt
  play vars_files
  role vars (defined in role/vars/main.yml)
  block vars (only for tasks in block)
  task vars (only for the task)
  role (and include_role) params
  include params
  include_vars
  set_facts / registered vars
  extra vars (always win precedence)

For roles you would generally use myrole/defaults/main.yml for easily overridable vars (e.g. installation path or smth) and use myrole/vars/main.yml for vars that you don't want to be easily overridden (smth like required packages).


K8s cluster in peered VPC can't reach RDS cluster

For my Peering Connection, I went to:
"Actions" -> "Edit DNS Settings" -> "Allow accepter VPC (vpc-<id>) to resolve DNS of requester VPC (vpc-<id-default>) hosts to private IP"
and checked it. (It was unchecked before). I had to wait a little while before this took effect.


How to revert the Jenkins setup if plugin/Jenkins updates went wrong.

As Tensibai mentioned in the comment - better way is backup you Jenkins instance before an update

There are different ways to do this:

Plugins

Jenkins has a lot of backup plugins, most popular is:

PeriodicBackup Plugin - be aware, has status This plugin is up for adoption right now, that means there is can be lack of support

thinBackup Plugin - the same about current status of plugin

SCM Sync configuration plugin - see attention on the plugin page, this is deprecated plugin, the suggestion about migrate their jobs to Pipeline Plugin and use the Configuration as Code Plugin also make sense

All of them easy to use, also you can discovery any Jenkins plugin here

Your own solution

Jenkins has good structure for backup, it is not hard to understand what exactly you need to backup and how.

You can write script\job with simple steps copy, archive, move to achieve backup of your Jenkins.

More information and discussion about how to backup jenkins can be found here: How do you back up Jenkins jobs & master configs?

As for me

I'm using Periodic Backup plugin + my own scripts for transfering backups into different places. But this plugin is not really stable and has opened issues, so i'm not recommend this for highload production use. That's fine for personal not-critical systems. Now I decided to migrate to my own solution with simple job to archive and copy all configs of my Jenkins instances.


How good are heartbeat, elasticsearch and kibana as a monitoring/alerting solution as a replacement for Nagios?

I haven't tried it myself, but in Alarms management with Elasticsearch and Kibana someone recommended Elastalert


How to implement the Load Balancing in Openshift ?

The simplest way is to find the icon that scales the application up and click it. In the current UI it's in Project --> Overview --> expand the application info (">" icon to the left), --> click "^" next to the blue circle to the right.

This gives you a default load-balancing across as many PODs (instances) of the application as your resources and imagination allow.

From the command line, this would scale it to 3 PODs:

oc scale --replicas=3 rc yourapplicationname

Another way is to deploy the same application with a different name, and then edit the route (Applications --> Routes --> RouteName) that points to it. There should be an option:

  Alternate Services / Split traffic across multiple services

This gives you control over how much traffic you want through each version. Good for A/B testing or canary releases.


Managing Data bags in different environments

https://www.digitalocean.com/community/tutorials/how-to-use-roles-and-environments-in-chef-to-control-server-configurations

  For instance, if a node is in the "production" environment, you could
  want to run a special recipe in your "nginx" cookbook to bring that
  server up to production policy requirements. You could also have a
  recipe in the nginx cookbook meant to configure special changes for
  testing servers.

Example

name "web_server"
description "A role to configure our front-line web servers"
run_list "recipe[apt]", "recipe[nginx]"
env_run_lists "production" => ["recipe[nginx::config_prod]"], "testing" => ["recipe[nginx::config_test]"]


How to automate TeamCity installation using docker?

Automation of installations like this can often be done with config management tools.  GitHub and the public repos for these tools is a good place to start.

There is a Puppet installer for TeamCity here: https://github.com/haf/puppet-teamcity

Here's a Chef cookbook:
https://supermarket.chef.io/cookbooks/chef-teamcity

You can also use tools like Packer to generate your image(s).  Packer can substitute for a Dockerfile in generating a Docker image. Packer will also run Provisioners (including Chef/Puppet/Ansible) to configure your image.


How does Vagrant differ from VirtualBox and similar software, if at all?

Vagrant is a tool to create and customize virtual environments (containers and VMs) in a reproducible fashion (across platforms) using code and a virtualization provider of your choice. VirtualBox is only one of the virtualization providers it supports. Vagrant can also create VMs using Hyper-V, VMware and AWS.

While VirtualBox's VBoxManage does let you create and configure VirtualBox VMs, Vagrant, by contrast, will allow you to create and configure VMs with the hypervisor provider of choice, which may or may not be VirtualBox. Vagrant is agnostic as to virtualization provider plug-in you use with it.

To sum it all up, Vagrant does not provide virtualization services itself.

Further reading

The links below provide helpful insights.

An itemized comparison of Vagrant to Hypervisors can be viewed on this HasiCorp github issue.

These links provide some additional overview of Vagrant that may help bring the difference into focus.

https://www.vagrantup.com/intro/index.html
https://en.wikipedia.org/wiki/Vagrant_(software)

Also, if you sign-up for a free account at LinuxAcademy you can view and work through this simple example that includes these tools.


How do I achieve Continuous Integration and Continuous Delivery (Ci-Cd) via JMeter?

Since you're talking about making delivery/deployment decisions based on the results provided by your script(s) you need to communicate with the team(s) responsible with maintaining the official CI/CD pipeline implementation for your organisation.

If they're using Jenkins and a particular pipeline implementation (multiple technological approaches are possible) compatible with the way you hooked up your script into Jenkins locally they may be able to directly use your approach.

Even if that's not directly possible, there probably still are ways of including your tests in the CI/CD pipeline, only with a more or less different implementation.

Note that there may be related technical and/or logistic questions/requirements that might need to be addressed depending on your organisation particularities, for example: ownership, development/maintenance process and policies, SLAs for the tests, script(s) and/or the resources for their execution.

Even if you don't get your tests integrated at first, you can still run them for every repository commit using your private Jenkins setup and make the results available for everyone to see. In time, if those results prove useful to others they may support your initiative and help with a reconsideration of the initial decision.


If you want to use environment variables with the ${VAR} syntax in the declarative pipeline, then you need to define those variables in the environment { } block.

for example:

environment {
        DISABLE_AUTH = 'true'
        DB_ENGINE    = 'sqlite'
    }

    stages {
        stage('Build') {
            steps {
                echo "Database engine is ${DB_ENGINE}"
                echo "DISABLE_AUTH is ${DISABLE_AUTH}"
                sh 'printenv'
            }
        }
    }

If you want to use environment variables that come from other plugins etc, then you need to access them with the env.VAR syntax.  For example:

echo "Running ${env.BUILD_ID} on ${env.JENKINS_URL}"


How are you enforcing git behavior, including locally (particularly on Windows)?

To answer the question of how to enforce something locally, you can't without doing some very heavy lifting around managing and enforcing the state of every developers workstation, and I'm usually of the opinion that developers should probably be local admins on their development machine because if they aren't then they will just spend their time figuring our how to get those privileges anyway.

And that's probably because you shouldn't need to care about the state of local configuration when using distributed version control. You should only care about the state of your configuration. Assuming you are using git as a centralised version control system, because that's what basically everyone does anyway because its easiest, then let's just assume that "your" configuration is the copy of code saved on the central server.

If this is the case then you shouldn't accept merges that, as you mentioned, break crlf/lf line endings. So you enforce that when some other client tries to push changes you with server side logic that rejects the request to pollute your repo with their potentially breaking stylistic choices.


Helm Installation fails with shared config maps

Couple of possible solutions:

Control the configmap from a third chart and add the third chart as a dependency in both the charts. This will duplicate the secret and can cause a skew in the values being used both the charts since if they have independent release cycles
Create the configmap in one namespace and refer to it using an external reference. You can see several examples of such at https://github.com/helm/charts/search?q=existingSecret&type=code

I personally prefer option 1 since there is explicit dependency specification and gives the ability to deploy the charts individually


terraform fmt

The terraform fmt command is used to rewrite Terraform configuration files to a canonical format and style.


How do you monitor and alert on thrashing containers?

Regardless if containers are ephemeral or not, there's two things you could consider not normal:

Non-zero exit codes
Restart count > 0 at the orchestration layer

How you can alert on those metrics will vary wildly between orchestration layers, providers and monitoring tool. I'll keep the discussion about the principles:

Non-zero exit codes

This one seems obvious enough. If you have an ephemeral workload, make sure it exits cleanly and you'll be able to monitor the health of your containers. Even if your ephemeral workload last 30 seconds and you launch thousand of them (small short-lived workers style), as long as everything exits cleanly, there's nothing to worry about.

Restart count at the orchestration layer

Most orchestration layers will expose a restart count. For Kubernetes, that's literally the 'Restart Count' metric when describing a pod. Those restarts are the orchestration layer restarting a container because it wasn't supposed to be in a 'stopped' state, but it was. Ephemeral workloads won't have a restart count, because of their very nature (or if they do, the "final" shutdown of the container won't cause a restart when the workload is done).

Whether you want to alert as soon as the restart count crosses a certain threshold or if you want to be more clever about it and calculate a rate (roughly: restart count / pod/service duration) is up to you.


How to monitor a micro-service oriented solution?

Prometheus is a popular tool for monitoring microservices deployed to the cloud in combination with Grafana. For a better understanding of your inter-service issues, a distributed tracing tool is needed as well.


How is it a good practice to version your helm charts for each deployment?

Approach you are describing - where app version is substituted at helm upgrade command - is pretty popular as well. Maybe it's not well documented - I'm not sure about that.

Yet another approach that people use is doing helm template which would compile everything (including variable resolution) into fixed k8s definitions and then doing GitOps on that.

There is no universally accepted approach here among those 3 (1st being updating helm chart itself with new versions every time). So you should decide what works better for your use case.

One thing that is worth mentioning is versioning issue (disclaimer: I'm solving it with Reliza Hub, which I develop). Basically, you need some system which would tell you what version tag of each microservice to actually use on different environments. That problem may become quite complex, especially if you have many microservices and many environments.


GitHub doesn't support VFS. There are news articles in 2017 saying GVFS and later VFS would be supported by GitHub. As of today, there is no support and no planned support for it.
My sources:

https://github.com/microsoft/VFSForGit/issues/1640
https://github.community/t/vfsforgit-setup/128184
https://github.community/t/microsofts-virtual-filesystem-for-git/2092
The Github public roadmap says nothing related to VFS or anything similar. https://github.com/github/roadmap


Halt execution of Bamboo build or deploy, without throwing an exception

You can manually stop a build in progress. From Stopping an active build:

To stop an active plan build:

Click Dashboard and then the All Plans tab.
Click the 'Stop' icon next to the active plan you want to stop.

It also appears possible to stop the build/deploy programmatically, using the REST API, more specifically making a DELETE request to the appropriate url:

/queue/{projectKey}-{buildKey}-{buildNumber : ([0-9]+)}?stage&executeAllStages
...
DELETE
Stop build execution.
available response representations:
401
Returned when user does not have sufficient rights to view or execute build for specified plan
204
On success (success is also when build was already completed - so nothing to stop)
404
Returned when specified job does not exist


Deploying software that depends on a virtual machine using Docker

Frankly, I've never seen running a VM inside Docker. I believe it would be very slow. Have you checked if https://github.com/budtmo/docker-android is useful?

This Video talks about the Docker and Android SDK as well - https://www.youtube.com/watch?v=YwBAqMDYFCU


Terraform: Provide credentials for docker hub registry

Docker registry credentials is seting up in provider configuration.

provider "docker" {
  registry_auth {
    address = "quay.io:8181"
    username = "someuser"
    password = "somepass"
  }
}

Also possible to set env variables DOCKER_REGISTRY_USER and DOCKER_REGISTRY_PASS


Are there serious problems with an EC2 auto-scaling AMI that automatically downloads newest server from git?

What you're going to do is only calling for inconsistencies between your nodes.
I wouldn't do any deployment during the instance boot-up, instead install AWS CodeDeploy as part of your boot-up (or bake it into the AMI) and once the instance boots up it contacts the CodeDeploy service and obtains the newest code version.
Likewise if you need to roll-out a newer code version push it to CodeDeploy and it will take care of updating all the servers in your Auto-Scaling Group in a consistent, predictable way.
With ASG you really really want to prevent inconsistencies between the nodes and this is one of the ways to do it.
Hope that helps :)


Found the answer to my own question, straight from travis support

  Hi Name,

  Thanks for writing in. Yes indeed, on our container based
  infrastructure, ICMP is not possible. Could you speficy sudo:
  required in the .travis.yml and see if it brings improvement?

  Thanks!

  Joep


According to the README of the deis/workflow project on GitHub, it is no longer maintained since March 1, 2018.

However, the README indicates that there is a fork, called Hephy Workflow. At the time of writing, this is the first time I heard about Deis and Hephy and I have not worked with these solutions.

Alternatives to these products are OpenShift, Google App, AWS Elastic Beanstalk, Azure App Service, Cloud Foundry, Apollo according to StackShare.

You mentioned that you are looking to build something internal. This would probably mean that native-PAAS like AWS Elastic Beanstalk, Azure App Service and Google App are not an option as these are only available on their platform.

One of the options could be OpenShift. This product has an enterprise and open-source version that can be installed on your on-premise or cloud servers. You could also decide to create a bunch of VMs on Google Cloud and install OpenShift on these servers.


Amazon S3 bucket does not redirect folder URLs to index.html

CloudFront does not actually have this feature. You can redirect the root object https://www.example.com/ to https://www.example.com/index.html by setting DefaultRootObject, but there's nothing for subdirectories such as https://www.example.com/posts/.

However, you can run arbitrary code on CloudFront using Lambda@Edge, so it's possible for us to add this feature ourselves. With the "origin request" trigger we can rewrite the request to S3, changing /posts/ to /posts/index.html. https://github.com/CloudUnder/lambda-edge-nice-urls has some good handling of paths that works well with Hugo.

Another option is to enable uglyurls in your config so that all links within you site have index.html hardcoded in to them.


Why are unnamed volumes created when `docker-compose up` is run and do these double after issuing `docker-compose down && docker-compose up`?

Research indicated that the anonymous volumes were created by influxdb and grafana/grafana.

  Anonymous volumes are not given an explicit name when they are first
  mounted into a container, so Docker gives them a random name that is
  guaranteed to be unique within a given Docker host. Besides the name,
  named and anonymous volumes behave in the same ways.

Results

version: "3"
services:
  influxdb:
    image: influxdb:latest
    container_name: influxdb
    ports:
      - "8086:8086"
    #env_file:
    #  - 'env.influxdb'
    networks:
      - backend
    volumes:
      - influxdb-data:/var/lib/influxdb

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    #env_file:
    #  - 'env.grafana'
    #  - 'secrets.grafana'
    networks:
      - backend
    volumes:
      - grafana-data:/var/lib/grafana

networks:
  backend:

volumes:
  influxdb-data:
  grafana-data:

results in:

DRIVER              VOLUME NAME
local               604a07040367512b09c618c6dcc71a7f55390c9c23de6ab08be7466414ed62da
local               7f4b630073b31b6e772d3edef6da81b48643525edfc34281ea13fbd6b86ec270
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data

and everytime when docker-compose down and subsequenlty up is run the number of anonymous volumes doubles.

What will happen if nginx will be pulled instead of influxdb and grafana?

DRIVER              VOLUME NAME
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data

and after docker-compose down && docker-compose up -d?

DRIVER              VOLUME NAME
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data

It looks like that certain images created the additional anonymous volumes. Let's replace influxdb with nginx and use grafana.

DRIVER              VOLUME NAME
local               15b80416ab06abb629d9f634a0feff08f7c560f31d614b9b430855c16cdb75c7
local               205a6f19cbf992c95b2e3be9f2fb1ca9ecec35fce550d0b7a4b9f32b0ef163b1
local               474108f5b7b14fba92a3e5a980f3bf851388b2ee25d7417df5c42d9f176e084b
local               5830a31a470ec8a42ddae7a37bb50487f3f36360318b2f9f5301b338507782b4
local               9f00868a2fec0cfc0d34dc12d0879d39487a13128863722f400ad4c47df2d340
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data
local               f47b1b7bbec8e50b32a7c39704c7c218165b284298d852313fa24bc7cbe6acc5

Everytime docker compose was run again, three anonymous volumes were created by the grafana/grafana docker image. Let's replace influxdb with nginx and revert the grafana to nginx:

DRIVER              VOLUME NAME
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data

and it remains two if restarted.

It looks like that grafana is causing the issue.

Why are three new anonymous volumes created everytime grafana/grafana is restarted?

The grafana/grafana dockerfile indicates that three anonymous volumes will be created:

VOLUME ["/var/lib/grafana", "/var/log/grafana", "/etc/grafana"]

https://docs.docker.com/engine/reference/builder/#volume

FROM ubuntu
RUN mkdir /myvol
RUN echo "hello world" > /myvol/greeting
VOLUME /myvol

  This Dockerfile results in an image that causes docker run, to create
  a new mount point at /myvol and copy the greeting file into the newly
  created volume.

docker volume inspect <volume name, e.g. 34cfafd4603dbc7e71a83e2520f978c8307b084143b3192de65a1995dc1d2f86> returned grafana data when the path that was returned was checked of two out of three anonymous volumes:

sudo ls /var/lib/docker/volumes/a0ecd00df8fc68ef36e777c7bf9ec5a496ee30e313b86889487501a53fa2e28e/_data
grafana.ini  ldap.toml

and

sudo ls /var/lib/docker/volumes/34cfafd4603dbc7e71a83e2520f978c8307b084143b3192de65a1995dc1d2f86/_data
grafana.db  plugins


Linking VSTS build with jenkins artifacts

So I wrote this powershell script to be ran by the deployment group agents who have access to the server locally.

Param(
[string]$JenkinsPatArg
)
$isRunningLocal = $false
$jenkinsUrlBase = "[public ipaddress]"

if(Test-Connection -ComputerName "[private ipaddress/DNS name]" -Quiet)
{
    $isRunningLocal = $true
    $jenkinsUrlBase = "[private ipaddress/dns name]"
}

$buildDefinitionName = $env:RELEASE_ARTIFACTS_GAMEZIP_DEFINITIONNAME
$buildTeamProject = [Uri]::EscapeDataString($env:SYSTEM_TEAMPROJECT)
$targetBuildNumber = $env:RELEASE_ARTIFACTS_[Artifact Alias]_BUILDNUMBER

$Auth = "$($env:username):$($JenkinsPatArg)"

$Bytes = [System.Text.Encoding]::UTF8.GetBytes($Auth)
$Base64bytes = [System.Convert]::ToBase64String($Bytes)
$Headers = @{ "Authorization" = "Basic $Base64bytes"}

Write-Host "Searching for $($targetBuildNumber) in project $($buildTeamProject)->$($buildDefinitionName)"

$correctBuild

Write-Host "Querying: http://$($jenkinsUrlBase):[port number]/job/$($buildTeamProject)/job/$($buildDefinitionName)/api/json?tree=builds[id,url,actions[buildNumber]]"

ForEach($build in (Invoke-RestMethod -Uri "http://$($jenkinsUrlBase):[port number]/job/$($buildTeamProject)/job/$($buildDefinitionName)/api/json?tree=builds[id,url,actions[buildNumber]]" -ContentType json -Headers $Headers).builds)
{

    if($build.actions.buildNumber -eq $targetBuildNumber)
    {
        $correctBuild = $build
        Write-Host $build.id
        Write-Host $build.url
        Write-Host $build.actions.buildNumber
        break

    }
}

if($correctBuild -eq $null)
{
    Write-Error "Build not found.  Must rebuild source to obtain artifact"
    return;
}

if($isRunningLocal){$correctBuild.url = $correctBuild.url.Replace("[public ip address]", "[local ipaddress/DNS name]")}

$targetZip = "$Env:temp/$($buildDefinitionName)_$($correctBuild.id).zip"

(Invoke-RestMethod -Uri "$($correctBuild.url)/artifact/$($buildDefinitionName)/*zip*/$($buildDefinitionName)_$($correctBuild.id).zip" -ContentType Application/zip -Headers $Headers -OutFile $targetZip)

Add-Type -AssemblyName System.IO.Compression.FileSystem;[System.IO.Compression.ZipFile]::ExtractToDirectory($targetZip, $env:targetDirectory);
#Expand-Archive -Path $targetZip -DestinationPath $env:targetDirectory -Force
Remove-Item $targetZip;

passing in the a personal API key (created by Jenkins user manager to use by visual studio team services), this script first checks if it is can access the server using local connections, if so set the variables appropriately.

From there we create local handlers for environmental variables.  Our release artifact name defined in VSTS is the same as that in our jenkins server.  The build number used is created by VSTS and passed to jenkins and stored by the VSTS plugin(Jenkins TFS Plugin download).  We get this version number from the environment as the artifact build number.  We also ensure that our team project path url is windows URL encoded brief explanation.  Next we hold our authentication which is stored as a variable in our VSTS variables.  The $JenkinsPatArg is marked as secret. As such, we must pass it in as it isn't visible as an environment variable.  We then encode the authentication and add it to our request headers.

We then add a debug line to print what target we are looking for and what project we are looking in, just to make sure we are looking for the right thing.  We create a variable to hold the build information that we find and do one more debug write stating what are query url is so that we can manually run it if we run into any issues.

Next we then run this url query against the build server and iterate over all the builds returned.  The query filters out all the JSON to just the build ID, it's absolute url, and the VSTS build number.  We check to see if the build we are examining has the same build number as the artifact we are deploying, if so store it in the $correctBuild variable, print the was found for debugging, and exit the loop.

Coming out of the loop we check to see if we found a result.  If we did not find a build (due to it being cleaned up most likely) we inform the user that the artifact needs to be rebuild from source.  Otherwise we download the artifact zip to a temporary location from the jenkins server and extract it to the target directory, defined as a variable.  Lastly we delete the downloaded zip.

This is ran as a powershell script task and due to it's length, can not be an inline script.  So to deploy this, i have a devops git repo for my scripts and I just include this as an artifact that is downloaded and ran.  This task is preceded by other tasks which stop the current process from running, and delete the current files in place before running this task.  After this task is ran, more scripts are ran to run the downloaded artifacts.

I hope this helps someone else trying to coordinate and link VSTS builds with Jenkins artifacts.


Maven dependencies that are downloaded not available in JFrog Artifactory server

In order Artifactory to cache your dependencies, you need to retrieve them from Artifactory. Once you use your virtual repository, which contains the Maven Central remote, Artifactory will download the dependencies from Maven Central and will cache them for the future use.

So, here's what you need to do in order to populate Artifactory with your dependencies:

Make sure that your virtual repository contains Maven Central or JCenter remotes.
Configure your Maven to work with Artifactory (The "Set Me Up" button in the Artifactory UI)
Delete the ~/.m2/repositories cache
Run your build again.


ansible aws windows ec2 - Internal Error: this connection module does not support running commands via sudo

According to the documentation:

  Prior to Ansible 2.4, become would only work when ansible_winrm_transport was set to either basic or credssp, but since Ansible 2.4 become now works on all transport types.

You're using Ansible 2.2. Might that be it? The error message might be a bit misleading.


I was able to find the root of issue. The problem was in the configuration of salt master. After every change of reactor config, I must run salt-call state.highstate command on master to sync reactor source code changes with it. I missed this step and thus my reactor config differed from my original source code in IDE. So once I ran mentioned command, everything became in sync. Thanks everybody for help!

P.S. regarding mentioned error, it occurred because I used kwarg in dev.sls for pillar and represented pillar as list:

>     kwarg:
>      - pillar:
>          temp_var_transmit: {{ payload }}

Kwarg can not contain a list in this case. It may contain key:value format only. So the correct way is either use without kwarg as I did or remove hyphen near pillar.


What software ihandles shipping to ElasticSearch the standard output of a .NET Core web application running in a container?

How do I send a structured message?

If you want structured messages check out some of the recommended third party logging options.  Some of these options have built-in support for Elasticsearch logging.  For instance, Serilog provides a "sink" for putting logs directly into Elasticsearch.

How do I configure it in code?

These options all tie into the ILogger interface, so testing different logging frameworks should require very little code changes.  However, the initial entry point of the application (e.g. Program.cs) will need to be configured to use the 3rd party logger.  The official .net core docs go over this in detail.

How do I configure it for Elasticsearch?

This will depend on the logger.  The appsettings.json will most likely be where you configure the .net core application's connection to Elasticsearch.


What are the cost effective ways to deploy services using Kubernetes on multiple deployment environments?

Based on my experience I'd suggest following scenario:
First you should make your apps deployment process similar for all environments, so same tool and set of files will be responsible for this part. And only difference should goes to parameters.
And second it's always a good practice to have kubernetes cluster per stage. Very common situation when your changes goes not only to kubernetes apps manifests, but to surrounding infrastructure also (LoadBalancer, Object Buckets, Disks Storage, etc).
You cat try to cut costs on kubernetes part by using same cluster for dev/stage but still situations with differences in infra resources could happen. And that could lead to more efforts on support and release deployment level.
In case you will use AWS, you can put kuberentes load to EC2 Spot Instances and that could heavily reduce your monthly bill.
As for toolset for k8s apps manifests, I've tried multiple tools and ways, and from my experience terraform+terragrunt works better here. Single tool for all infra resources, parameterization works easily, common code could be separated to reusable modules.
Other option is kustomize or helm, but those tools could help with kubernetes manifests only, and can't manage external resources.


I had expirienced the same problem, I needed to renew manually my PAT.

VS

MSDN


Preventing an IAM user from overriding launch template parameters on RunInstance

Alternatively, is there another way I could allow an IAM user to launch an instance in line with a template, while preventing them from making changes (such as choosing a super-expensive instance type)?

I've encountered this problem last year and have circulated up to our AWS account's architect, who contacted their IAM division.

AWS's response is that this is intended behavior. Reason given was that "An instance-type spec on a launch template is not a resource, or conditional that can be used to granted/deny permissions to". Notice that inside the IAM policy, specifying a launch template is a condition to action "RunInstance", and the resource of said policy is an ec2 instance.

According to AWS IAM team, in order for you to restrict instance-type (a condition on the resource, "instance"), you must specify the restriction as part of the condition clause to be applied to the instance, like the following:

{
  "Effect": "Allow",
  "Action": "ec2:RunInstances",
  "Resource": "*",
  "Condition": {
    "ArnLike": {
      "ec2:LaunchTemplate": "arn:aws:ec2:ap-southeast-2:xxxxxxx:launch-template/lt-xxxxxxx"
    },
    "Bool": {
      "ec2:IsLaunchTemplateResource": "true"
    },
    "StringNotLikeIfExists": {
      "ec2:InstanceType": [
        "c5.*xlarge",
        "p3.*xlarge"
      ]
    }
  }
}

Obviously, it is near impossible to list all of the expensive instances we don't want our users to use, and kinda defeats the purpose of using a launch template.  But in a messed-up way, in the IAM world AWS created, this actually makes sense.

Consider the docs here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ExamplePolicies_EC2.html#iam-example-runinstances-launch-templates

Note the language: The policy uses the ec2:IsLaunchTemplateResource condition key to prevent users from overriding any of the launch template *resources* in the RunInstances request.  So that clause only applies to resources (like NICs, IPs, etc), but not instance-type, which is not considered as a resource.

I asked that they revise their docs to explain this, and of course, nothing's changed.


Is there any concrete and acceptable solution for running systemd inside the docker container?

You didn't mention what distribution you're using inside your container (which would have implications w/r/t which version of systemd you're using), but the following will successfully boot a CentOS container running systemd:

docker run -it --rm \
  -e container=docker \
  --tmpfs /run \
  --tmpfs /tmp \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --cap-add SYS_ADMIN \
  centos /sbin/init

This is with Docker 17.05.0-ce; older versions may require additional flags.  Using a stock centos:7 image, your initial environment inside the container looks like this:

# systemctl status
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2017-12-04 17:47:15 UTC; 45s ago
[...]

And:

# ps -fe
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 17:47 ?        00:00:00 /sbin/init
root        16     1  0 17:47 ?        00:00:00 /usr/lib/systemd/systemd-journald
dbus        26     1  0 17:47 ?        00:00:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root        28     1  0 17:47 ?        00:00:00 /usr/lib/systemd/systemd-logind
root        30     1  0 17:47 console  00:00:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
root        32     0  0 17:47 ?        00:00:00 bash
root        58    32  0 17:48 ?        00:00:00 ps -fe

Note that I'm using --rm here not because it's necessary but because I'm terrible at cleaning things up after the fact.  It's not necessary to get the container to run.

  but looks like most of them compromise the security of the container and the host

Well, running systemd does require privileges beyond those granted to a typical Docker container (hence the --cap-add).  Whether this has security implications for your environment or not depends on what you're doing.


Can Terraform S3 remote state be kept in a different AWS account?

You can have two approaches:

Assume the role that has access to the AWS bucket. Assume this role in every account (don't forget to set up trust relationship). This is your approach. I think the issue might be some objects (state files) were created in a different way. Other culprit can be an account-specific KMS key that is used to create the objects. Don't forget, KMS keys are region-specific.
You grant access to other “users” via the S3 UI. (At the time of writing it is not supported by the Terraform AWS Provider.)

Go to the S3 bucket -> Permissions -> Access Control List -> Access for other AWS accounts -> Add account
Find the canonical ID for the account you want to give permission to.
Set the bucket-level and object-level permissions. Beware, the Principal (owner) of the object will be different, so the account that actually hosts the bucket might not have access to it.


Is there any latest technology which will replace Docker, Kubernetes in future?

Perhaps serverless could substitute it, but it will be hard to predict. I would suggest to follow industry standards and run things using these tools in production to stay up to date. The most changes in all branches occur in IT. Having said this, every engineer has to learn new things everyday to stay up to date. Whether k8s and docker will be replaced? Probably yes, but on the other hand there are still VMs and on-premise hardware. Perhaps k8s and docker will be used less in the future, but when I do not know. I am not a fortune teller with a crystal ball.


Jenkinsfile script to waitUntil a folder is created

Your block expression doesn't return a value in all cases.

Try this instead:

waitUntil { folder.exists() }


RUN fails while installing Angular on Ubuntu Image

Finally managed to generate the docker image for Angular. Here are is the screnshot.

I am listing the contents of the Docker File along with the comments that explain the reasoning and my learnings.
# Created by Sridhar Pandurangiah - 29 October 2020
# This Docker file helps create the containers for the Angular User Interface
#1 We would like to create from the node image and then add the layers. I had initially started from a base image and proceeded to add Ubuntu, it did not work out as creating a ubuntu from the base involves a lot of work and its rather baseless to start from a base image. Then switched to starting from the Ubuntu image and the problems I faced is documented in this question. So had to start from the node image. Also remember that docker containers have to address a single concern.

FROM node:15.1.0 as ab-suite-image

#Add Information about this image
LABEL maintainer="sridhar@st.in"
LABEL version="1.0"
LABEL decsription="This is a docker image built from the node image and includes angular9 and metronic theme"

#2 Add Angular and other peer dependencies that do not get installed by default to avoid the build from exiting with errors
RUN npm install -g @angular/cli@9.1.12 @angular/core @angular/common eslint chokidar jquery popper.js tslib --save

# Set environment variables for file locations if required

#3 Set the environment Variable to determine the environment where the application is deployed development | testing | staging | production
ENV AB_COMPLIANCE_ENV=development

#4 Set the working directory to copy the angular dependencies
WORKDIR /opt/ng

#5 We don't pull the code from GIT as it would mean storing the private ssh keys on a docker image that can be publicly accessed. This is is not advisable.
COPY package.json ./

#6 Update project dependencies using package.json. This stage is where we encountered a lot of issues with build failing as the shell scripts were returing errors. the flag --legacy-peer-deps ensures that the npm doesn't throw errors so the docker build continues
RUN npm install --legacy-peer-deps

#7 Copy all the other project files and build
COPY . ./
RUN ng build

#8 Configure Volumes if required

#9 Copy start Script if required
#COPY start.sh /start.sh
#CMD ["./start.sh"]

#10 Expose Port for the application
EXPOSE 4200 443

This docker image can be used to for running unit and E2E tests.
Some literature on the web ask you to install nginx but not advisable. Run nginx on a seperate dedicated container. On production you can build the angular application with the "prod" flag and copy the files so that nginx can serve them.


Can you use `aws-sdk` to store objects in Google Storage?

Most of the AWS SDKs mentioned support constructing a custom endpoint, for example as described here for Node.JS.

Setting the endpoint to storage.googleapis.com and using your GCP access credentials should enable you to operate the Google storage backend service "like S3".


Why unable to find the exact path for grains storage in master?

These are built-in grains, which are collected at minion startup. Here's the code responsible for generating them.

You cannot remove them, but you can overwrite them, for example to an empty value in /etc/salt/grains:

osfinger: null

or with grains execution module: salt minion1 grains.delval osfinger.

Keep in mind that some of the core grains are essential for the correct functioning of salt, e.g. module providers for pkg module rely on them.


What is the difference between SRE and DevOps?

DevOps is about maintaining code, is that correct?

DevOps is not "just" about code, or systems, or any one thing.  DevOps is a very general term that covers all things related to software delivery.

Site Reliability Engineering is a term popularized by Google. From this article https://landing.google.com/sre/interview/ben-treynor.html we can distill their TL;DR:

  Fundamentally, it’s what happens when you ask a software engineer to
  design an operations function.

Operations, Engineering, and Software Development are blurring together.  The degree of automation required to create and maintain a mature infrastructure requires skills from all three.  SREs are admins, and engineers, and developers.

See also:
http://shop.oreilly.com/product/0636920041528.do


amazon ec2 ip whitelisted on a third party service provider

A possible approach could be:

Put your instances in a private subnet
Route incoming traffic through a load balancer
Route outgoing traffic through a NAT

Outgoing traffic will then use the elastic IP assigned to your NAT as its source IP.
An added benefit of this approach from a security perspective is that your instances won't be directly reachable from the internet, thus reducing your attack surface.


How can Jenkins help to implement CI/CD for Drupal?

What you've listed is actually two different things, continuous delivery and continuous integration.  When people talk about CI they typically are only referring to the testing part; automated unit, function, and integration testing with some sort of version bump or code promotion at the end.  Looks like the Jenkins wiki has some specific pointers about Drupal.

As far as environment promotion, you're getting into continuous delivery which Jenkins can also be used for.  Consider looking at the more modern Jenkins Pipelines for CD pipelines.


Q: "Is it possible to change ansible temp dir for one playbook only? I have VMs with r/o root."

A: Yes. It's possible. Configure the variable ansible_remote_tmp.
In some cases, a better solution to this problem is ANSIBLE_PIPELINING. Quoting from Risks and limitations of become:

Use pipelining. When pipelining is enabled, Ansible does not save the module to a temporary file on the client. Instead, it pipes the module to the remote python interpreter’s stdin. Pipelining does not work for python modules involving file transfer (for example: copy, fetch, template), or for non-python modules.

Notes:

Quoting from Release notes:

The ANSIBLE_REMOTE_TMP environment variable has been added to supplement (and override) ANSIBLE_REMOTE_TEMP. This matches the spelling of the config value. ANSIBLE_REMOTE_TEMP will be deprecated in the future.

remote_tmp is not among configuration options in Ansible Configuration Settings

See remote_tmp parameter of the sh plugin.

There are open bugs mostly with Windows


Unable to ssh in a multi-machine vagrant environment

Following Vagrant file address this problem.

You can get all supporting key files along with this vagrant file at https://github.com/malyabee/IaaC/tree/master/ansible_lab

$commonscript = <<-SCRIPT
sudo yum update -y
sudo yum install python2 epel-release -y
sudo yum install -y ansible
sudo echo "192.168.22.10    ansiblecontroller.example.com ansiblecontroller" >> /etc/hosts
sudo echo "192.168.22.11   node01.example.com   node01" >> /etc/hosts
sudo echo "192.168.22.12   node02.example.com      node02" >> /etc/hosts
SCRIPT

$nodescript = <<-SCRIPT
cat /vagrant/ansible_lab.pub >> /home/vagrant/.ssh/authorized_keys
SCRIPT

$ansiblescript = <<-SCRIPT
sudo yum install ansible -y
sudo cp -r /vagrant/ansible_lab /home/vagrant/.ssh/id_rsa
sudo chmod 400  /home/vagrant/.ssh/id_rsa
sudo chown vagrant:vagrant /home/vagrant/.ssh/id_rsa
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.provision "shell", inline: "echo Hello"

  config.vm.define "ansiblecontroller" do |ansiblecontroller|
    ansiblecontroller.vm.box = "centos/7"
    ansiblecontroller.vm.provider "virtualbox" do |v|
          v.memory = 512
          v.cpus = 1
       end
    ansiblecontroller.vm.network "private_network", ip: "192.168.22.10", virtualbox__intnet: "mynetwork01"
    ansiblecontroller.vm.hostname = "ansiblecontroller.example.com"
    # Installing required packages for ansible controller node
    ansiblecontroller.vm.provision "shell", inline: $commonscript
    ansiblecontroller.vm.provision "shell", inline: $ansiblescript
  end

  config.vm.define "node01" do |node01|
    node01.vm.box = "centos/7"
    node01.vm.provider "virtualbox" do |v|
          v.memory = 512
          v.cpus = 1
       end
    node01.vm.network "private_network", ip: "192.168.22.11", virtualbox__intnet: "mynetwork01"
    node01.vm.hostname = "node01.example.com"
    # Installing required packages for  node01
    node01.vm.provision "shell", inline: $commonscript
    node01.vm.provision "shell", inline: $nodescript
  end
  config.vm.define "node02" do |node02|
    node02.vm.box = "centos/7"
    node02.vm.provider "virtualbox" do |v|
          v.memory = 512
          v.cpus = 1
       end
    node02.vm.network "private_network", ip: "192.168.22.12", virtualbox__intnet: "mynetwork01"
    node02.vm.hostname = "node02.example.com"
    # Installing required packages for  node01
    node02.vm.provision "shell", inline: $commonscript
    node02.vm.provision "shell", inline: $nodescript
  end
end


How do I get more control over the NodePorts my Service assigns to my Deployment?

As far as I understand kubernetes services, that trick is not possible.

What could help you to have preserved hostnames is a StatefulSet.
See more here

  The Pods’ ordinals, hostnames, SRV records, and A record names have
  not changed, but the IP addresses associated with the Pods may have
  changed. In the cluster used for this tutorial, they have. This is why
  it is important not to configure other applications to connect to Pods
  in a StatefulSet by IP address.


According to the Jenkins Job DSL documentation shallow has to be enabled:

shallow(true)


Service to service communication - through proxy or direct?

The question is: why use a proxy to communicate both applications? What is the point or target (support SSL, security, availability between services?
In any case, you need 2 proxies, one in each app, in front of each application, it is the best way to use it.
You can also use tunnels, like SSH tunnel or Stunnel that support TLS


Using Jenkins to control docker image build flow

I have found a solution so I'll leave it here for others.

using docker

#!/usr/bin/env groovy

pipeline{
agent buildsvr

  stages{
    stage('Checkout and build Docker Source'){
      steps{
        checkout([
          $class: 'GitSCM',
          branches: [[name: '*/master']],
          doGenerateSubmoduleConfigurations: false,
          submoduleCfg: [],
          userRemoteConfigs: [[
            credentialsId: 'asdf787c-5fd4-4dfb-81ad-873jhf107223',
            url:
'git@mydockersourcerepo.git']]])
        sh 'pushd docker && docker build -t ${image}:latest . && popd '
      }
    }
    stage('Checkout Target Source'){
      steps{
        script{
          checkout([
            $class: 'GitSCM',
            branches: [[name: 'mybranch']],
            doGenerateSubmoduleConfigurations: false,
            extensions: [[
              $class: 'SubmoduleOption',
              disableSubmodules: false,
              parentCredentials: false,
              recursiveSubmodules: true, reference: '',
              trackingSubmodules: true
              relativeTargetDir: 'source']],
            submoduleCfg: [],
            userRemoteConfigs: [[
              credentialsId: 'Bitbucket-Jenkins',
              url: 'ssh://git@sourcerepo.git']]])
        }
      }
    }
    stage('Build Source Target'){
      steps{
       script{
          sh 'docker-compose up'
        }
      }
    }
  }
  post{
    success{
      script{
        sh 'docker tag ${image} 10.90.239.223:5000:/${image} && docker push
10.90.239.223:5000:/${image}'
      }
    }
  }
}


Chef resource to register gitlab runner automatically

Your second link just use the official cookbook. This official cookbook still uses the LWRP syntax, so you need to check in the provider directory for the resource action code -- which for register is basically just two execute commands (extract from the file linked above):

converge_by("Register runner #{new_resource.description}") do
  execute "#{env} gitlab-runner register --non-interactive #{options}"
  # giving time to gitlab-ci to manage the request
  execute "sleep #{new_resource.sleep}"
end

So mostly, your exemple ends up calling the gitlab-runner register command with the options defined in the options hash (with a little transformation of the hash into command parameters).

There's no 'core' chef resource to register gitlab runners out of using an execute resource to run the command.

A basic example of this resource is present in the cookbook Readme:

gitlab_ci_runner 'my runner' do
  options({
    registration_token: '1234567890',
    url: 'http://gitlab-ci.myinstance'
  })
end


Who is responsible for running and maintaining shared services in DevOps

I think we need to view this in the larger context of Flow and Systems Thinking which is one of the core values of DevOps.

Specifically, DevOps Values require an understanding that each individual is part of a larger system and work toward increasing the flow of value through that system to the customer.
DevOps also focuses on helping the organization set a shared vision and work incrementally and collaboratively on achieving that vision.

In such a scenario, the focus needs to shift from "technology-specific silos" to "cross functional teams". And each DevOps practitioner needs to achieve a T-Shaped Skillset with core as well as shared responsibilities.

So, to answer your question:

  Does DevOps really mean you are using another service so you have to manage that service as well?

I think it depends.

If your being able to manage the service leads to increased flow of work through the pipeline to the customer then yes.

If however, it leads to you becoming a bottleneck or constraint to flow (perhaps due to you becoming overloaded with additional responsibilities), then no.

If you are able to share this responsibility with others than manage the service (and have a Systems and Flow mindset), and collaborate with them so that none of you get too overloaded with work to become a constraint to flow, then yes.

References:

DevOps Values
YBI/YRI
What is DevOps
Cross-Functional Teams and DevOps


Terraform - resource repeated multiple times

Terraform scans the configuration directory for all files whose name end in .tf, and loads them all as configuration.

A common cause of this problem is if you accidentally duplicate one of the .tf files under a new name, and do not update its contents. In that case, Terraform will see both files define the same resource and produce the error shown here.


How do I make NFS volumes reliable in Docker?

There are options in docker that will retry the startup of a container, restart in version 2 and restart-policy in version 3 (you'd need compatibility mode enabled for the version 3 syntax to work). However, I believe they only work when the issue is from the application inside the container fails, not when there is an issue creating the container like you see with a volume mount failing (or would also happen if the image couldn't be retrieved from a registry).

To handle the failing volume mount, I believe swarm mode is your best option despite your objections. You can create a single node cluster with docker swarm init and deploy your compose file with docker stack deploy -c docker-compose.yml stack_name, making an easy transition from docker-compose. Swarm mode looks at the overall state of the service and continuously tries to make the current state match your target state (as defined in the compose file), which will handle a failing volume mount that eventually corrects itself. I don't have an NFS server to test on right now, but here's a scenario with a bind mount to a missing folder:

$ cat docker-compose.vol-bind.yml
version: '3'
volumes:
  bind-test:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /home/bmitch/data/docker/test/missing
services:
  bind-test:
    image: busybox
    command: tail -f /dev/null
    volumes:
      - bind-test:/bind-test

$ docker stack deploy -c docker-compose.vol-bind.yml voltest
Creating network voltest_default
Creating service voltest_bind-test

$ docker service ls
ID                  NAME                   MODE                REPLICAS            IMAGE                                             PORTS
omzaeo7mrour        voltest_bind-test      replicated          0/1                 busybox:latest

$ docker service ps omzaeo7mrour
ID                  NAME                      IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR                              POR
TS
kpz0l79eucaw        voltest_bind-test.1       busybox:latest      bmitch-asusr556l    Ready               Ready 2 seconds ago
j6fylzhvcv60         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 5 seconds ago    "starting container failed: er…"
61o6raohp0xl         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 12 seconds ago   "starting container failed: er…"

$ docker inspect kpz0l79eucaw
[
    {
        "ID": "kpz0l79eucaw1856obmwvcak1",
        "Version": {
            "Index": 445
        },
        "CreatedAt": "2019-04-29T12:57:25.925788528Z",
        "UpdatedAt": "2019-04-29T12:57:34.3467203Z",
        "Labels": {},
        "Spec": {
            "ContainerSpec": {
                "Image": "busybox:latest",
                "Labels": {
                    "com.docker.stack.namespace": "voltest"
                },
...
        "Status": {
            "Timestamp": "2019-04-29T12:57:33.936048295Z",
            "State": "failed",
            "Message": "starting",
            "Err": "starting container failed: error while mounting volume '/home/var-docker/volumes/voltest_bind-test/_data': failed to mount local volume: mount /home
/bmitch/data/docker/test/missing:/home/var-docker/volumes/voltest_bind-test/_data, flags: 0x1000: no such file or directory",
            "ContainerStatus": {
                "ContainerID": "4c75c851bd43b5ef57b4785a4611f78501279933822eab90c693e1108f53ee82",
                "PID": 0,
                "ExitCode": 128
            },
            "PortStatus": {}
        },
...

$ mkdir missing

$ docker service ps omzaeo7mrour
ID                  NAME                      IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR                              POR
TS
okylli7smu1c        voltest_bind-test.1       busybox:latest      bmitch-asusr556l    Running             Running 4 seconds ago
9tpm188ysu2k         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 14 seconds ago   "starting container failed: er…"
kpz0l79eucaw         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 21 seconds ago   "starting container failed: er…"

From the above, you can see as soon as the missing directory was created, the bind mount succeeded, and the container that failed to be created as retried and started successfully.


Why isn't my S3 Cross Region Replication working for some files?

So I verified it was definitely public access causing the issue somehow; uploading an identical file but with the public read acl on totally broke it.

I worked with AWS support and they noticed I had "Block public access" enabled in the destination bucket.  Apparently rather than replicating the object and then making it private, it just fails replication.

tl;dr if you want to replicate objects that are public read, don't block public read at the destination.


Github plugin triggering to push event only for changes in specific branches

Finally I have resolved it with the help of "strategy of choosing what to build" with "Default behavior" for each branch.


How to Versionize Ansible (Tower) with GitHUB

one strategy is to put your playbook repo(s) thru a regular build/deploy pipeline, where you separate between "nonrelease" and "release", for example by linking the master branch to "release" (most CI tools support this out of the box). Then you'd have two "streams" of artifacts you could deploy, one for release and one for non-release (or release candidate or whatever you'd like to call it). These can be auto-deployed to different folders inside Ansible tower and then picked up as different "projects". That way youd have two "instances" of every playbook folder (or "project" as Tower calls them).


If it doesn't outright fail you can expect to see a cycle warning where you've created a loop in Terraform and it's trying to fulfil your request for asg-name before the asg has been created.

To get what you need you'll need to have something like the following:

module_apps/module_apps.tf

resource "aws_autoscaling_group" "example" {
  name = "${var.asg_name}"
}

module_init.tf

variable "asg_name" { default = "asg-ad" }

module "module_init" {
  source = "../module_apps"

  asg_name = "${var.asg_name}"
}

output "asg_name" {
  value = "${var.asg_name}"
}

This way, if you call module_init somewhere else you'll have asg_name available to you. Always be careful of trying to use an output of a module to create the module, Terraform doesn't handle loops like that well and the errors can be confusing to diagnose.


How do you monitor varnish cache inside a docker container?

Varnish is monitored with what they call VSM. It's a file in your container. If you bind mount the folder where the files is, you can share the same folder / files in other containers that will be able to read it, and thus will be able to monitor your varnish instance.
See the note about VSM and containers in varnish docs.
I have been able to launch a varnishstat with statistics in a container with varnish in the image, but no varnish running, like this:

docker run --rm -it \
       --entrypoint=/bin/sh \
       -v /var/lib/docker/overlay2/$hash/merged/usr/local/var/varnish/$hostname:/varnish \
      emgag/varnish:5.1.3 varnishstat -N /varnish/_.vsm


Assign Directory Read/Write Privileges to Azure Service Principal from the command line?

What you need here are the az ad app permission commands.

Here is a complete example (assuming powershell):

$apiId = "00000002-0000-0000-c000-000000000000" # Windows Azure Active Directory
$apiPermissionsId = "78c8a3c8-a07e-4b9e-af1b-b5ccab50a175" # Directory.ReadWrite.All
$spn = az ad sp create-for-rbac --role="Owner" --name blah | ConvertFrom-Json
az ad app permission add --id $spn.appId --api $apiId --api-permissions ("{0}=Scope" -f $apiPermissionsId)
az ad app permission grant --id $spn.appId --api $apiId
az ad app permission admin-consent --id $spn.appId

For reference, Microsoft has a list of the GUIDs used for various applications.

On some occasions I have been unable to find documentation for the magic GUIDs required for a particular app/permission, in which case I would resort to adding them through the portal manually (Azure Active Directory ➔ App Registrations ➔ Required Permissions) and then reading the JSON manifest back to reveal the required GUIDs which can then be plugged into automation.


Creating a cron job with Ansible (ERROR! 'cron' is not a valid attribute for a Play)

crond.yml in your example is not a playbook. It is a tasks file (list of tasks), while playbook is a list of plays.

A play must have hosts directive at minimum. Please see http://docs.ansible.com/ansible/latest/playbooks_intro.html#hosts-and-users

This is a playbook:

- hosts: localhost
  tasks:
    - cron:
        user: "root"
        minute: "30"
        hour: "02"
        job: '/usr/bin/rsync -avhz -e "ssh" --rsync-path="sudo rsync" -H --delete --numeric-ids <ipaddress>:"/data/rsnapshot-backups/<ipaddress>" /data/rsnapshot-backup'
        state: present


Does Ansible have component akin to PuppetDB?

The direct answer is "no".

Ansible does not have any database, because it is supposed to be run by different operators from different computers and it's purpose is to be sure that target systems are in the exact state, as specified in playbooks.

There's an option to attach external facts cache backends to Ansible. So you can setup redis as your cache, Ansible will populate it with gathered facts during playbook run and you can later query redis for them. But it is very limited solution, because there are not too much facts gathered by Ansible, for example there are no information about installed packages – you will have to gather them yourself an use set_fact with cacheable option to put them into external cache.


Condition to skip a test when running in a container

In Ruby you can use this function:

# return true if we are inside a docker container
def in_container?
  return File.file?('/.dockerenv')
end


Should local Ansible playbooks be loaded each day/week with cron?

I would discourage you from using Ansible to run Ansible (though this can be done) as you will run the risk of breaking idempotent rule of Ansible. Instead, choose a singular starting point for running your playbooks; either command line, cron or some other program.

With that said, I would use cron to run an Ansible playbook, however I would also setup your ansible.cfg file for logging and check the log periodically if there is breakage in your playbook.

Ansible by default doesn't do logging so I would set this up in ansible.cfg, just in case:

[defaults]
log_path = ./ansible.log

Also, make sure to check your email regularly to make sure your cron/Ansible job ran without any problem.


Using docker-compose, what's the recommended way to handle different sets of containers for dev and prod?

docker-compose.override.yml, docker-compose.prod.yml and extend

An outline answer would be to have a docker-compose.yml file for your common base configuration, a docker-compose.prod.yml for production-specific sets of containers and for other specific environments, docker-compose.override.yml. It is possible to use the docker-compose extend keyword/command/statement in a docker compose yml file to inherit from another docker compose yml file.

The above approach is on the basis that one would first consider only the things common across all environments as the pure, base setup and then dev and prod environments would be based off of this (inherited).

Your dev-specific packer setup could be configured in the docker-compose.override.yml file, which inherits from the base docker-compose.yml. Configuration that is common across all environments would reside in docker-compose.yml. Your production setup could be in docker-compose.prod.yml, which again can inherit from the base docker-compose.yml.

References

This seems to be the best practice approach, based on the following references:

The inheritance/nesting approach is explored here: Inheritance or
nesting with docker compose
The official documentation here - Share Compose configurations
between files and projects - describes the docker compose
extend keyword.

Some things you may or may not need to consider for your specific case

There may or may not be some bespoke config code required in
addition i.e. in your deployment itself, e.g. bash scripts etc., should the above approach provide most of what you need but not quite everything. These scripts can be run from docker compose.
Research what can be overridden when writing docker-compose.override.yml
when "inheriting" from docker-compose.yml. Your particular project may require a configuration in each of the dev and prod environments that override the same thing in the base environment, thus the need to override what is in the "parent"/base docker-compose.yml environment. Consider this issue: https://github.com/docker/compose/issues/3729 - which suggests there are limitations as to what can be overridden, particularly if you want to move a configuration item that was present in the parent docker-compose.yml. A better way could be to only have items common across all environments in docker-compose.yml and have the specifics in files for specific enviroments, such as docker-compose.prod.yml


Not really. The standard way to work around this though is to use eg:

terraform apply -target=aws_security_group.my_sg

but that's only going to apply one security group at a time, so will get tedious if you have a lot of them. You can, however, target multiple resources in one command:

terraform apply -target=aws_security_group.my_sg -target=aws_security_group.my_2nd_sg

However, there are potentially a couple of workarounds:

The -target parameter respects dependencies.

This means if you were to eg. -target=aws_instance.my_server and that instance had, say, five security groups attached to it via interpolation, changes to those security groups should be included in the plan (I haven't thoroughly tested this, but I believe this is how it works).

That is a bit messy though, as you probably don't want to touch an instance. A safer alternative might be using something like a null_resource to provide a target for the security groups, but again I haven't tried this (there might be another 'safe' resource you could rely on, though?).

Create a module.

You can target a module just like you can target a plain resource:

terraform apply -target=module.my_security_groups

Inside this module, you could define all of your security groups - just like you would have outside of the module. As well as being able to target it directly, this also makes it easier for you to re-use the same set of security groups for other infrastructure, if you ever need to.


How to check out GitHub repo after specifying skipDefaultCheckout in Jenkins declarative pipeline?

You can use the checkout scm step whenever you need the source:

pipeline {
  agent none
  options { skipDefaultCheckout() }
  stages {
    stage('Build') {
      agent { node { label 'builder' } }
      steps {
        checkout scm
        echo 'build-the-app'
        stash(name: 'app', includes: 'outputs')
      }
    }
    stage('Test') {
      agent { node { label 'tester' } }
      steps {
        unstash 'app'
        echo 'test-the-app'
      }
    }
  }
}


How to tell kubectl to use different api groups

It works in dotted notation (Object.apiVersion.APIGroup), for e.g. deployments this way: kubectl get deployments.v1.apps In your case just try:

kubectl get issuer.v1alpha2.cert-manager.io -n=default


Accessing TPM from inside a Docker Container

I think that the statment "to store the Vault's data in TPM" is not valid.  The TPM is generally not for storage.  There are some slots for storing some values and likely some persitent keys, but it is only a handful. I think TPM 2.0 maybe has more storage than 1.2, which I'm more familiar with, but it is not a storage device.  It really depends on what and how much of it you need to store.

So while there might be a solution using "on-chip" TPM memory, it is scarce and really slow. The more common pattern is the use the TPM to wrap the keys or data, such that it can be stored in an encrypted blob on actual storage, like disk or network, and only safely unwrapped on THAT MACHINE.

The TPM is not really a "crypto co-processor" nor is it a storage device. It's slow and has minimal resources, for cost.

Having clarified that, it probably is valid to use the TPM, in the proper way, to help securely store the application data, it just depends on how much work it is for you.

Also, if the data doesn't need to be specifically tied to that specific machine, or that specific container, or the installation is on a VM, and the VM might spin up on different hardware, then using the TPM doesn't really make sense because it is a root-of-trust for a specific platform, a specific board or machine.

So to be semantically picky:

Storing the Vault data in the TPM: not valid
Storing the Vault data with the TPM: valid (potentially)

Disclaimer:  I don't know anything about Vault or the data you want to store.


Override Ansible task parameters from role

Defining default value for ansible variables and overriding them via ansible playbooks can be achieved using Jinja2 templating.

- name: Create a new user
  user:
    name: "{{ username_variable | default('default_value') }}"
    password: "{{ password_variable | default('default_value') }}"
    state: present

Then if these variables are passed from the playbook it will be used, or else the default values.

  roles:
    - role: common
      vars:
        username_variable: "frank"
        password_variable: "/bib/zsh"


Most answers to these questions are going be subjective and influenced by your particular organization's workflows and culture. Little of this response if more than just my own opinion and experience and much is not really specific to Azure DevOps but can be applied to any build and release orchestration tooling.

Azure DevOps itself has very little to do with your chosen workflow apart from it being the common remote for the team and the source for pull requests etc. What source workflow you use is highly dependent on your individual situation. The number of people on your team doing simultaneous work, how much possible conflict there is between feature development efforts, your rate of release, complexity of code/projects, and philosophies about release management to name a few. Much has been written about various source management workflows. In the realm of Git, of course there's the over 10 year old GitFlow approach (note the recent update) and GitHubFlow. I won't regurgitate what they or the numerous other writings suggest about what flow to use. Your team needs to research the options and evaluate what is most applicable.

It certainly is not a bad practice to do a build for every PR. If developers are following good practices where feature branches originate from the merge target, there is a high probability that any testing/compiling they did in code from those branches will essentially match what will eventually be merged. However, that's never a guarantee so having the ability to do a pre merge PR build is a very helpful feature of ADO. And ADO can be configured to required a passing PR build before merge. You can also deploy PR builds if you want to support initial manual testing and/or more complex integration of a PR prior to merge.

This again will be a choice driven by your team and situation. Do you want every feature merge to result in a build? If you are practicing Agile, you may have work items that require manual testing as part of the definition of done. In that case, getting builds out immediately after code is complete could be imperative to forward progress. I follow a continuous integration approach and have builds trigger immediately upon commits/merges to the "typical" branch of work in progress (often that will be the development branch). The sooner you discover a problem, whether that is a compile error of the merged code, a unit test failure, or a manual integration test bug the better. Every minute you wait to fix an issue is compounded into more time resolving it later as the work fades from peoples' minds.

I have typically followed the philosophy that "what I deploy to production should be what we tested". While theoretically, if you do 2 builds of the same exact source code you should get the same result, there's no guarantee of that. Particularly now with the use of public packages/modules and the ability to allow flexibility with what versions are consumed, it's not an unreasonable possibility that a later build will result in different artifacts, for example, but pulling in a newly updated version of a dependency. For some very real perspective on this, see this article about how the "internet broke" some years ago. I work in the healthcare industry which has very strict rules that drive our implementation approach. We need to deploy just what was tested and approved to the production environment. If we did a fresh build for production release that introduced a changed library with an unknown security vulnerability, we could get into serious hot water.  I follow the practice of having a build pipeline do the actual build of final artifacts which become the singular source for deployments for testing and final promotion to production. ADO makes this very simple by publishing the artifacts into its own internal storage and providing those as an artifact source for a release pipeline. You can then deploy that (and additional) artifacts to each release stage of your pipeline. One key factor to consider is environment configuration management. Depending on your deployment target, you'll need to take into account how to influence the runtime configuration. My organization still deploys .NET applications to traditional IIS web servers using file copy. This requires an explicit step to apply config transformations for the application based on the target environment. This approach can be enhanced by using a separate artifact source for the transformation to better implement the good DevOps practice of separation of concerns (environment agnostic build artifacts vs. env specific configuration; env configuration, including secrets, can be retrieved from a different, more protected source so they are not in the application code repository). If you are deploying to a docker image, your docker runtime environment can drive environment specific settings for the container instances. Deploying to a cloud application host service? Use its instance configuration tooling, such as the app service/slot specific configuration settings in Azure. Applications like React, however, take a different approach that typically dictates that you do a build for the target environment since environment driven values are bundled into the deployed artifacts. I'm not a fan of this as it breaks my core philosophy of "build once, run anywhere". But there are some solutions to help make such builds environment agnostic.


Ansible does not create directories CentOS7

You have two mistakes:

name as parameter of file module. It is an alias of path, so you actually try to create directory add dir.
Unnecessary single quotes in path, giving you actual paths like etc/ansible/roles/webservers/'handlers'.

Try with:

...
- name: add dir
  file:
    path: '/etc/ansible/roles/webservers/{{ item }}'
...


is a running container created by each RUN command during multistage build?

Yes your understanding is correct. The RUN <commands> command in the Dockerfile is somewhat equivalent to
container = $(docker run base-image) #1
docker exec base-image commands #2
docker commit $container #3

Since images are static, we'll need a running instance/process to execute our command. this is why a container is created from the base image. #1
After the execution of the command in #2, the changes are committed back #3, which means a new layer is created on top of the previous layers. This layer will contain the changes introduced by the container in (#2).


Is there a DevOps methodology aimed towards academia and students? No.


When is it ethically acceptable to fork someone else github code and continue with that?

It is always ethically acceptable to fork any code on Github (or Sourceforge or wherever you got it from) and do whatever you want with it, within the terms allowed by its license! This is what open source and permissive licenses are all about. The original author wanted the world to contribute to this piece of software, and you will be doing exactly that.

Fork it, respect the license requirements, and go on with it. It does not matter whether it is abandoned (like your example, since the last change was 3 years ago) or current.

If you do feel that you did some good improvements, send pull requests back to your upstream, as  usual. If the original author never comes back, then so be it. If he does, he can integrate them (or not if he so chooses). But this is strictly optional, you are neither legally nor ethically forced to do so.

About uploading it to some other locations (Ansible Galaxy or whatever - say Docker Hub for docker images and so on): the issue is probably mostly to avoid irritations due to the name being "replaced" (but I'd assume if the original author already has uploaded it to A.G., you can't just replace it, anyway).  So, in the latter case, just upload it with another name, which is still easily identifyable as related to the original - say, append some suffix which is not too obnoxious.

If the author, in your case, should later re-appear, and you had uploaded the software to the Ansible Galaxy, and he wants to take over, then you two will simply communicate and resolve the issue somehow (i.e., merge the two forks again and decide who keeps updating Ansible Galaxy). Chances are that he will be more than glad to leave the work to you.

Oh, and to be sure; this piece of software is under the MIT license:

  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

There you go.


How to run multiple commands with command module in Ansible

The Ansible documentation states that the command module doesn't get a shell

  The command(s) will not be processed through the shell

The command module takes the command as an argument, so you can't have  list as you've written there. You could do the same thing in a loop: with command: {{ item }} :

- name: "Run {{ item }}"
  command: "{{ item }}"
  loop:
    - cmd2
    - cmd2
    - cmd3

You could also have several command: tasks instead.

However, this is generally considered bad practice (E.g. E303 from Ansible Lint rules) because it tends to break idempotency. It would be better to decompose those commands into something that uses Ansible modules.


Unable to use the class defined in src in vars of jenkins shared library

It looks like Groovy can't find your class at the path specified.

Try renaming the file common.groovy to match the class name and move it to the common folder:

src/org/common/CommonFuncs.groovy


How to calculate disk space required by Prometheus v2.2?

I think your math may be off.  If you use the ~18600 number you found I get a very different result from 68TiB:

2,592,000 (seconds) * 18600 (samples/second) * 1.3 (bytes/sample) = 62,674,560,000 (bytes).  This is ~62.675 Gigabytes (divide the digital storage value by 1e+9
) and I'm guessing a more reasonable number for your infrastructure.


Host credential manager and Docker runners

You need to provide more information about the technology stack.
In a nutshell, you would mix the responsibilities of the host, taking care for the gitlab runnner + credential store is not a good design practice. If the host goes down, you are loosing 2 essential services, instead of just one. Most likely you would have other consumers of the store. Also this is breaking the single responsibility principle.
I could tell you from own experience is that we use 'AWS Parameter store' for that  ( H/A and detached service ). The secrets are consumed by a helm chart - https://github.com/cmattoon/aws-ssm and injected into the cluster as configurations. After that each interested party could use whatever it wants.


Provide customers with access to Azure Container Registry

There is a balancing act here between the simplicity and maintainability of your solution, and how auditable creating separate service principals or users for each of your external customers.  I suspect only you can actually decide which is the best practice in your specific legal and regulatory situation.

On the side of having a single Service Principal with multiple passwords, there is a minimal amount of configuration required and other than perhaps limitations imposed by Microsoft on the number of authentications per second or the number of passwords that can be created for a Service Principal it should work quite nicely.  The downside is your resource logs won't give you any information about who pulls images, you will have some logs about issuing a short-lived password, but you can't prove that's related to a docker login.

On the side of creating a Service Principal per client means your resource logs will tell you who pulls images.  However, you will need to automate the creation and maintenance of those Service Principals to account for clients joining and leaving.

Why does it matter? it may not for you, however if one of your customer's machines was compromised and the images leaked onto the internet you may want to hold one of the customers to account.


There is no need to push any OTP values into the keycloak database.
You can use the new privacyidea keycloak provider.

The user logs in with his username/password and the privacyidea-provider will take care, that the users get the email and ask in a 2nd dialog for the OTP.

See:
https://github.com/privacyidea/keycloak-provider


Docker in docker volume mount doesn't work

Single quotes in shell meant to leave the enclosed string 'as is'. That is the variable inside single quotes isn't going to be evaluated. Try changing '${WORKSPACE}' for "${WORKSPACE}".


Is there a way to combine helm with some other tool for k8s configuration management without conflicts?

Helm isn't made to be used this way.  If you'd prefer to use a static deployment, you're better off using a plain Kubernetes manifest.  Then, people changing the image field won't break your release.
You can easily convert it to one simply by running helm get manifest command
Helm is used to manage release versions.  I.e. every new release (and a docker image change definitely qualifies as one), you would push a change to your helm chart.
The other alternative you have is defining docker image as a value.  Then you can pass a command-line argument to replace the image tag with a new one and re-run helm from Jenkins any time you have a new deploy.
Example:
in values.yaml:
image:
  repo: foo
  tag: bar

On command line when running helm:
helm upgrade --install my-app --set image.tag=newtag

Final, and least recommended way is to use a mutable tag (i.e. "master") and set your ImagePullPolicy to Always.  This way, when you have a deploy, you just need to bounce all the pods in your deployment/daemonset/statefulset object.


tools to gather variant information from thousands of servers and update into database

Doing something with collectd feeding into Prometheus or datadog is definitely a good idea for gathering metrics, but it doesn't help you with things like the kernel version.  Handling process monitoring might be better handled with nagios or datadog as well.  But for the bulk of what you're doing I'd say that it makes the most sense to stick with ansible.
What you're doing sounds very similar to the ansible-cmdb project:

Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page (and other things) containing system configuration information.
It supports multiple types of output (html, csv, sql, etc) and extending information gathered by Ansible with custom data. For each host it also shows the groups, host variables, custom variables and machine-local facts.

I've been using this for the last couple of years and it continues to evolve and improve.  I hope you'll find it useful as well.


How to automate control of synchronization between Maven artefact version and Git tag?

then I can tag the codebase whatever I want like v.0.1

One could use semantic versioning, i.e. X.Y.Z. instead of v.0.1


How to perform logging of Ansible crons in Kibana?

Ansible is an automation tool, so it can automate things you can do by hands.

But as far as I know no existing cron-like services support sending logs to elasticsearch (Kibana is just a viewer, btw) out of the box.

So at first, you should create some wrapper script that will execute your actual job and send it's result to elasticsearch.

An then schedule this wrapper script via Ansible.

Or choose another route, setup syslog forwarding to logstash/elasticsearch on your target box to collect all logs, then just filter your desired cron jobs.


Jenkins DSL - Pipeline Job (Unexpected token '$')

I have just managed how to escape characters.

The solution is to add: "\" before the character.

checkout([\$class:....


When do I run applications on a server in a pod and when to run them directly on the base operating system?

We're using a self-hosted gitlab, nexus and Jenkins (controller) and both are hard installs on CentOS7. Jenkins agent is a remote CentOS7. That approach made the most sense to us as opposed to breaking out the whole automated pipeline shebang to deploy gitlab/jenkins/nexus. Gitlab/Jenkins/Nexus are part of the foundation of our CICD infrastructure so it made no sense to make them ephemeral by using container, scaling or orchestration approaches. The actual software components we build & deploy are ephemeral (ie: a new build / config & deploy many times a day) and that infrastructure building, deploy & config is pretty much automated from scratch.


dotnet not found on travis for an F# project

OK - found the solution myself. It's a kind of tricky one. Even though my project is written in F#, I still have to set

"language": "csharp"

in my .travis.yml file. That's a bit counter-intuitive, but I must admit that it's written in the doc, so can't really blame anyone.


AWS S3, as an organizer see empty bucket

Does the bucket have versioning enabled? You should be able to see it in the bucket properties: Versioning: enabled. If so, when you are looking at the empty folder, at the top there is a switch "show versions", switch it on. Do you see some objects with hashes?
Try to delete them. Be careful, there are versions and "delete markers", you have to delete both in order to completely empty the bucket.
Another option: you are billed for requests and data transfers https://aws.amazon.com/s3/pricing/?nc=sn&loc=4
If something is checking the bucket rapidly, it can generate some costs.
But most often the delete versions is the reason.


How to allow a non-root user to write to a mounted EFS in EKS

If anyone comes across this later I resolved my issue by using an initContainer for any Pod that needed to write to the file system.
For example:
apiVersion: v1
kind: Pod
metadata:
  name: app3
spec:
  containers:
  - name: app3
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out3.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
    securityContext:
      runAsGroup: 1337
  initContainers:
  - name: fs-owner-change
    image: busybox
    command:
    - chown
    - "root:1337"
    - "/efs-fs"
    volumeMounts
    - mountPath: /efs-fs
      name: cog-data
  securityContext:
    fsGroup: 1337
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: efs-claim

The rest of the definitions match what was in my question.


Is it possible to set role in FreeIPA to use for applications as Jenkins, Github?

Based on your title, no.

It's impossible to quantify if every application has this capability. Many do.
Users/Groups in Jenkins OSS map back to LDAP but you control the permissions via Jenkins.

As you can see above Users/Groups have specific permissions that are specified/controlled in Jenkins. If this group exists in LDAP (xxx) the users in this group get the permission you setup in Jenkins. This makes it easier then managing individuals.