how to run Docker/Jenkins behind a proxy

I believe that the problem lies on your proxy environment variables.

$ docker run \
    --name myjenkins \
    -p 7000:8080 \
    -p 50000:50000 \
    --env HTTP_PROXY="http:// localhost:8080" \
    --env HTTPS_PROXY="https:// localhost:8080" \
    -v /var/jenkins_home \
    jenkins

The values of $HTTP_PROXY and $HTTP_PROXY are not supposed to have space characters on them and the protocol should be http. Additionally, although setting $HTTP_PROXY is useful for compatibility with other applications running inside the container, curl expects the HTTP proxy environment variable to be written in lowercase, that is, $http_proxy. From curl (1) manual page:

  Environment

  The environment variables can be specified in lower case or upper case. The lower case version has precedence. http_proxy is an exception as it is only available in lower case.

  http_proxy [protocol://]<host>[:port]
  Sets the proxy server to use for HTTP.

  HTTPS_PROXY [protocol://]<host>[:port]
  Sets the proxy server to use for HTTPS.

  FTP_PROXY [protocol://]<host>[:port]
  Sets the proxy server to use for FTP.

  ALL_PROXY [protocol://]<host>[:port]
  Sets the proxy server to use if no protocol-specific proxy is set.

  NO_PROXY <comma-separated list of hosts>
  list of host names that shouldn't go through any proxy. If set to a asterisk '*' only, it matches all hosts.

Therefore the docker run command might be written as below:

$ docker run \
    --name myjenkins \
    -p 7000:8080 \
    -p 50000:50000 \
    --env HTTP_PROXY="http://localhost:8080" \
    --env http_proxy="http://localhost:8080" \
    --env HTTPS_PROXY="http://localhost:8080" \
    --env https_proxy="http://localhost:8080" \
    -v /var/jenkins_home \
    jenkins

However, my solution will certainly not work because the port 8080 inside the container is being used by Jenkins and not by a HTTP proxy. Please, could you provide more information on how your work proxy is set up so I can edit my answer with more accurate details? Are you running a HTTP proxy such as squid or cntlm in your computer that receives requests from applications running in your workstation and forwards them to the corporate proxy after authentication? If so, you will need to adjust container's network configuration according to Stack Overflow question From inside of a Docker container, how do I connect to the localhost of the machine?.

EDIT: According to comments, a cntlm instance running locally on port 8080/tcp provides a HTTP proxy service that local applications must use in order to access internet. I believe that the least complex way to expose it to Jenkins container is by specifying --network=host parameter and sharing the root network namespace with the Jenkins container. However, Jenkins is set to listen the port 8080/tcp by default, therefore it must be instructed to listen to an alternative port, such as 7000/tcp. That can be achieved by passing --httpPort=# command line argument to Jenkins via JENKINS_OPTS environment variable.

EDIT 2: While analyzing the repository code, I've realized that Jenkins is a Java application and I am not sure that JVM recognizes $http_proxy and $https_proxy environment variables properly. So, to ensure that Jenkins will use cntlm as HTTP proxy, I also suggest passing proxy configuration properties to the JVM via the JAVA_OPTS environment variable.

EDIT 3: The Docker image jenkins is deprecated in favor of the jenkins/jenkins:lts one. The former image is currently unmaintained and seems to contain a version that is outdated and incompatible with plugins being downloaded.

To sum up, the docker run command have to be written as below for Jenkins to work in your environment:

$ docker run \
    --name myjenkins \
    --network=host \
    --env HTTP_PROXY="http://localhost:8080" \
    --env http_proxy="http://localhost:8080" \
    --env HTTPS_PROXY="http://localhost:8080" \
    --env https_proxy="http://localhost:8080" \
    --env JAVA_OPTS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=8080 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8080" \
    --env JENKINS_OPTS="--httpPort=7000" \
    -v /var/jenkins_home \
    jenkins/jenkins:lts

Passing --env JAVA_OPTS="-Djava.net.useSystemProxies=true" instead might work. I am not able to perform a try, though.


How to properly automate CM based on log events?

There is a natural tension here between:

(a) fixing something that is wrong automatically: because for example, a CVE has been created with a high enough severity to need to do something about it.

(b) breaking the system: because the automation introduced a breaking change or a defect.

I have a tendency towards systems being under constant development, even if no features are being added to them they need to kept current to avoid exposure to vulnerabilities.   The rate of change to the 3rd party components of the majority of systems I cover is so high that's its actually worth rebuilding everything and redeploying it daily.

This introduces a couple of requirements:

It is highly automated,
99% of user journies are covered by end-to-end tests so each build has a high degree of confidence that it will work,
We don't roll out to all users, we employ canary release and a rollback mechanism,
We use version constraints in package manifests, etc... i.e. the version of React must be >= 16.7.1 and < 17.0.0
We manage by exception, i.e. if the end-to-end tests fail or the canary release fails humans get involved.

This, in turn, makes systems like trivy, OWASP ZAP and SonarQube more about generating metrics and stopping vulnerabilities from going live than triggering automated remediation.

We do then have a set of rules like:

The live application must not have an unmitigated CVE with a CVSS score of 7.0 or higher.
The list of CVEs with a score of 4.0 or higher will be reviewed on a weekly basis to assess if remediation should be undertaken (i.e. change to a different library, introduce a new safeguard, remove a defunct library, fix the CVE at source, etc.)

This then gives my SRE teams the operational boundaries to work within, they may in practice:

just re-run the deployment ad-hoc to fix it now rather than within 24h,
override a default, for example, if there is a known vulnerability deploy something with a less critical but still higher than CVSS 7.0 known vulnerability,
Introduce a new safeguard, i.e. a Web Application Firewall Rule to mitigate,
Swap out the dependency for something less vulnerable,
Disable that functionality entirely until a fix can be introduced.


How to run two react project through nginx on same machine

Try to stop the react applications and see if the message changes - and tbh i think you should see 502 if nginx was not routing properly


Pooled Build Server and Multibranch development to stabilize the trunk?

Attempting to run your validation on branches, before being integrated into the trunk won't get you very far. The reason is that fundamentally such validations are performed in isolation, most often they will not actually reflect the trunk picture after the branch is merged.

The only case in which the branch validation is, well, valid, is if no other change (of any kind) makes it into the trunk between when the validation started (assuming the latest trunk changes are merged into the branch at that time) and when the branch is merged back into the trunk.

If any other change makes it into the trunk while the validation is in progress it won't be taken into consideration, leaving room for breakage when the branch is merged into the trunk. To prevent that you'd have to merge again the trunk into the branch - to pick up the latest trunk changes - and re-run the validation.

Effectively this means the branch validations (or at least the successful ones allowing the branch to be merged into the trunk) need to be entirely serialized if you want to keep the trunk stable. Which may be manageable for a while if you keep a reasonably small number of such branches (maybe multiple devs working on the same branch?). How many branches? Depends on how stable you want the trunk to be: if you want to always pass your integration tests, your serialized validations would need to run the integration tests. At 40 min each you can support max ~60 branches (assuming the merges are automated and can be performed 24x7, which isn't trivial). Without automated merges the number of branches will be a lot smaller. This won't scale, the branch lifespans will gradually increase, diverging more and more from CI and will eventually wear down your team.

The same reasoning explains why devs shouldn't actually have to run all tests if they are too lengthy - by the time they complete them the results may be invalidated by other changes committed to the trunk. Also - results tests done by developers are often not that relevant ("But it built fine in my workspace!").

There are only 2 possible approaches (that I know of) to get relevant validations:

the ones performed by the traditional CI tools - performed after the changes are committed to the trunk, reflecting the actual trunk reality, with all changes integrated. The only problem is if these validations fail - effectively blocking further development on the trunk until the culprits are found and the regressions fixed. Which may be (very) difficult in larger scale projects - as you already noticed.
the ones performed by gating CI tools - only 2 out there that I know of: Zuul and ApartCI (which is my baby). These validations are performed prior to merging changes into the trunk, but in an orchestrated manner, ensuring they remain relevant - taking into consideration all other changes in flight that will eventually be present in the trunk by the time a particular change is merged into the trunk. The actual change merging into the trunk is entirely automated, executed by the CI tool itself, immediately after the relevant validations are completed successfully. These are, IMHO, the only ones capable of guaranteeing trunk stability in larger scale projects.

Pooling resources is a good thing - give them to the CI tool - that's how you maximize their usage.


Its very common to provide a StatusPage (like Statusy.co or StatusPage.io). There are numerous examples of major providers having them:

https://status.github.com/messages
https://status.bitbucket.org/
https://status.aws.amazon.com/
https://azure.microsoft.com/en-us/status/
https://api.twitterstat.us/

You could provide the same status page to your customers. When there is a service disruption, you could manually update the status while providing detailed, customer-facing information. Or you could hook it into your Zabbix monitoring to have the status auto-updated.

This also lets your customers see a history of incidents and average uptime of your service.

There are also self-hosted, open-source solutions that you could implement:

https://cachethq.io/
http://www.stashboard.org/

Although, you'll really want to host those outside your existing infrastructure so that your StatusPage can always be reached, even if you're down. If you go the self-hosted route, I recommend hosting the status page at a different provider: Digital Ocean, AWS, etc...


If you're running this code today via script console, you're probably running into a RejectedAccessException from the sandbox which is why you require admin access to run this coupled with and or needing administrator access to use the script console.

If you put this code into a shared library, the code becomes trusted, and you no longer require administrator access to run the code and you no longer need script console. You just need a job that can call the shared library, if needed you can restrict access to the folder or job.


How to identify the physical IP of where Kubernetes API server runs

The apiserver runs on master nodes. To get a list of all the master node IP's, this should help-

kubectl -n kube-system get po | grep -i apiserver | cut -f1 -d" " | xargs kubectl -n kube-system get po $1 -o=jsonpath="{.items[*].status.hostIP}"

Based on the version of Kubernetes you are running, you might need to update the location of the hostIP in the jsonPath parameter.


Where do I set that agent certificate for Puppet on Amazon Linux?

On the agent I needed to set the server ip set to puppet in
/etc/hosts

I had accidentally put it on the agents ip address and then been incorrectly assuming i'd done something wrong on the master rather than the agent.

(server ip)  (domian name)   puppet


Does K8s ClusterIP service supports custom timeout?

All cookies and headers are a construct of the HTTP protocol. Kubernetes Services operate on TCP/UDP packets therefore it will pass the payload as-is.

It is not that there isn't an nginx.conf file but rather the configuration is managed via annotations. See this link
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md


The term Alert Fatigue, also known as Alarm Fatigue has been around for decades. It affects many professions, including medical, technical, and construction industries.  According to the Alert Fatigue by Other Names reference, the earliest literature on the topic came during the Israeli Arab conflict.  The practice/understanding of the issue is ancient.

No, PagerDuty did not invent the term.

Alert Fatigue:

  occurs when one is exposed to a large number of frequent alarms (alerts) and consequently becomes desensitized to them.

Aesop's Fable about the boy who cried wolf is a classic example of Alert Fatigue.  In Technology and Medical fields it occurs when alarms/alerts occur so frequently (with or without reason) that we become desensitized to them.  The outcome of this desensitization:

  dramatically reduces our reaction to similar subsequent threats.

In other words we are become so used to the alert that we either ignore it or take our time to acknowledge it.

References

Alert fatigue by other names
Alarm_fatigue
Cry Wolf - When Experience Becomes Fateful


Can I create an Azure VM from Ansible with NICs in different resource groups?

From close reading of the Ansible doc page on creating Azure VMs, it's clear that you can refer to a resource such as a virtual network or NIC using its resource ID.  This lets you specify any resource, just like RM templates.

To specify a NIC created in another resource group by the networking team, with the networking_rg_name var set to that group name, you would use a resource ID like this:

- set_fact:
    nic_id: "/subscriptions/{{ subscription_id }}/resourceGroups/{{ networking_rg_name }}/providers/Microsoft.Network/networkInterfaces/{{ nic_name }}"

- name: "Create VM {{vm_type}} - {{name}}"
  azure_rm_virtualmachine:
    resource_group: "{{rg_name}}"
    name: "{{name}}"
    vm_size: "{{size}}"
    admin_username: "{{user}}"
    admin_password: "{{pass}}"
    os_type: "{{os_type}}"
    network_interfaces: "{{nic_id}}"     # Use the nic_id instead of name
    image: "{{image}}"
    tags: "{{tags}}"

This works for any resources that are in a different resource group, including virtual networks.


Well that's not exactly about chef, you're basically asking how to use tar.

The command to create an archive of a folder would be
tar -zcvf <backup_thing>.tgz <path_to_backup>

In the tar options:

z pass the resulting tar archive to gzip for compression
c instruct tar to compress (create the archive)
v tells tar to be verbose, so you know at which point it is working, it has a performance impact
f <path> tells tar to output in a file or device

A correct chef resource for this would be as follow:

execute 'Backup #{dir_name}' do
  user 'user'
  command "tar -zcvf #{dest}/#{dir_name}_bkp_#{timestmp}.tgz #{dir_name}"
  live_stream :true # to see the progress during chef run, remove it and v in tar option if you don't care of it.
end

Using an execute resource avoid spawning a bash process to run tar, it is not needed, the shell_out class used by execute under the hood handles it already.

Be warned this is not idempotent, each run of chef will redo this backup command, that's maybe not what you're after.
For an update of jenkins (or something else) out of usual maintenance I would do a manual backup, check it is ok, and then run the upgrade (via chef eventually).

I'd recomend taking the tutorial at https://learn.chef.io and reading through the documentation at https://docs.chef.io


run ansible command as another user using their ssh key

become_user will tell which user to become to run a particular ansible module on the remote server.

If you want to run the ansible command on your control machine as an other user, you need to use sudo or su on this machine e.g.

sudo -s -u ansible ansible all -m ping

The -s option will load the user's shell an make sure homedir and all other envs are correctly set prior to running the command.

An other possible solution would be to use the ssh key you created for the ansible user from your account

ansible all --private-key=/home/ansible/.ssh/id_rsa -m ping

This will potentially work only if your current user has read access to the given key.


How to correlate Devops deploy on demand with Agile Sprint Process

There is no "Deploy-once-a-day" rule in devops philosophy. It's more of: Deploy as soon as possible and as often as possible. Also it calls for decoupling architecture so different parts of it may be released separately and also for decoupling deploys from release.

The Deveops Handbook by Gene Kim et al.calls for modifying Definition of Done to include running in production-like environments. That is follow the devops principles all the time during the sprint.

What you want to avoid is ending a sprint with all the developers handing over their code that is tested, is working but is not integrated and is handed to other team to be deployed, somehow.

I'd argue that definition of done is : it's not done until it's ready to run in production. Once it's ready you may or may not release it depending on your specific business needs and the sprint goal but if it's not ready to be released right now it's not done.


What are the Devops-related conferences happening around the world?

Velocity Conference is one of the most popular Devops conferences, and also is an O'Reilly conference.

From it's website:

  At Velocity, systems, site reliability, and systems engineers, systems
  architects, and application developers learn to build and manage fast,
  resilient, and highly available websites and apps.

The videos of the conference are present on O'Reilly's youtube channel.


Git clean/smudge filters for ansible vault secrets

The problem here is caused by the random salt in ansible-vault encryption. You can hack VaultEditor class to pass the salt to it from an argument in ansible-vault. The random salt is generated in lib/ansible/parsing/vault/__init__.py on this line. It is called from lib/ansible/cli/vault.py where you can easily add the argument for fixed salt. If you do change it, please submit an upstream patch to Ansible, I'd love to use it.

This problem is further discussed here on hacker news. And there are other implementations with tools that take fixed salt, namely gitcrypt, transcrypt. Here is also a link to one more implementation using ansible-vault called ansible-vault-tools, but this one has the same salt issue as far as I know.


How to ensure EC2's OS/application is loaded?

What you are talking about is known as a health check. Typically these are handled by load balancers, to decide whether instances are "healthy", and thus can serve (usually HTTP) traffic. The load balancers will poll and endpoint that you define to decide whether the instances are healthy, usually by seeing if the HTTP response code is some known-good one, like 200.

One solution therefore, could be to front your application with a load balancer, and enable health checks on the relevant urls. This will check whether your application is serving traffic properly, when polled by the load balancer. While this adds a level of complexity (and cost) to the app, it is also quite easy to do.

Of course, you could mimic this functionality without an AWS load balancer, by having your app poll the url that the app is supposed to respond on, by using the DNS or IP name returned by AWS after the instance becomes "running".

Another option would be to set up an actual monitoring system to check whether the application is serving content. A very simple and free tool that would do the job is e.g. Freshping.

You could also write an Inspec Compliance profile to describe a known good state of the machine. Inspec can connect to the machine via SSH, which you specify as being open. This may sound like overkill, and quite similar to the ICMP test you considered before, but it's nevertheless an option :)


How do you fetch a Jenkins job's artifacts from a remote system?

The public URLs of artifacts can be found in Workspace of the job. It is advised to use view version of URL, so it is more consistent.

For archived files, Archived Artifact Url Viewer can provide URL to a file inside a zip or jar archive within the artifact folder of a build


WorkItem Tracking error when performing clone operation in Azure Devops

This is an indication that someone removed a field from work item configuration that was represented by 25.  It's more of an issue of clean-up not being done when fields are removed.  See more about this error on Microsoft's Developer Community.

This other item in the community suggests a different problem when using a particular extension and a fix has been supplied.


How to forcibly remove the resource created by the broken module?

The workaround is to pull the current state file, edit it and use it as a base line. For example:

$ terraform state pull > terraform.tfstate
$ vim terraform.tfstate # Carefully remove invalid entries.
$ python -mjson.tool terraform.tfstate # Validate JSON.

Note: In Vim, placing cursor on the opening bracket, hitting d% will remove the whole group.

Then plan and apply:

$ terraform plan -state=terraform.tfstate -refresh=false
$ terraform apply -state=terraform.tfstate -refresh=false

Alternatively, pull, edit and push your local state file into remote state:

$ terraform state pull > terraform.tfstate
$ vim terraform.tfstate # Remove invalid entries and increase the serial value.
$ terraform state push terraform.tfstate

For Azure, it's better to use terraform-provider-azurerm instead of general DNS provider (terraform-provider-dns).


Make use of CIs as Maven Repository in order to use it for building Docker images without including dependencies

Docker images are after all VM templates, i.e. they have to be more or less self-contained: you get the image, you run the environment with all the dependencies.

To approach the challenge operationally, Docker supports reuse of environments through the FROM statement i.e. you could maintain a base image and on top a much smaller image with the app itself.

But, from the strategic point of view, I would say your architects might want consider splitting the whole thing into smaller microservices - that would allow for 5 smaller images. Of course I cannot judge whether it's really possible.


Restart docker container from another container

In majority of circumstances I wouldn't recommend this in production, but to restart docker container from another container, you need to install docker in that container and then run that "master" container with

docker run -v /var/run/docker.sock:/var/run/docker.sock

as described here: https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ (look for "The solution" part of the article).

But check your architecture and see if you really need to restart one container from another or if there is some other way.


A "Walking Skeleton" is a form of "proof of concept" of your basic architectural concept. Where a proof of concept typically focuses more on a single functionality, a "Walking Skeleton" is a minimalistic end-to-end implementation. A "Walking Skeleton" is not an outline of your concept (only a "skeleton") but is really executable and shippable (it can "walk" :O) and should be accompanied with tests.

Alistair Cockburn has described it (and is being quoted often):

  A Walking Skeleton is a tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel.

The advantage here for DevOps is that a "Walking Skeleton" should be developed early on in the project and results in working, shippable and testable code. This way DevOps can set up a full continuous integration chain early in the project, instead of being onboarded in the final phase of projects. This means that any issues that would arise are also being tackled in an early stage instead of rush work at the end.


What options are available to report the build status of Jenkins Multi Pipeline's first branch build on k8s to bitbucket-cloud?

As I was not able to tackle the bug in the Java code and the bug was already reported in 2017 I decided to write this plugin. This binary is able to send a build status even when a branch is built for the first time.


Make container start if docker daemon or host system is restarted

You need to add a restart policy to your container.

example:

docker run -dit --restart=always hello-world

In this example, I am telling docker to always restart the hello-world container no matter what happens (and also run it in the background interactively). If you do a docker ps, you will see it keeps restarting. To revert this, you need to update the container restart policy like this:

docker update --restart=no <CONTAINER_ID>


How to get hostname from node-exporter in Docker Swarm?

You just have to use the hostname on your docker compose file like this:

hostname: whatever-{{.Node.Hostname}}

Note that you have to prepend something to the hostname to avoid hostnames duplicates.


How to serve a custom index.html using the stock nginx docker image?

The "official" nginx image has a specific path where it looks for files. You can either copy a file into the image by creating a new image from it or mount your files into this path as a volume.
To copy files, have a folder with your index.html and a Dockerfile that looks like this:
FROM nginx
COPY index.html /usr/share/nginx/html

Then build a new image with docker build -t my-nginx .
To start a container with your own files as a volume, you can run the container like this:
docker run -p 8080:80 -v ${PWD}:/usr/share/nginx/html nginx

All the files in your current folder (including the index.html) will now be available via http://localhost:8080/index.html.
Most of this information is covered in the documentation for the image at https://hub.docker.com/_/nginx


How to deploy loki-stack helm chart with CRIO?

Managed to solve this by changing the values.yaml in the loki-stack helm chart to the following:
promtail:
  enabled: true
  pipelineStages:
  - cri: {}

Enjoy.


Trade offs of putting an international master database in us-west-2

It's been a while since posting the question, and a comment I made above seems to be helpful despite the fact it's using a quantitative assessment, so I'll post it here as an answer.

As per comment above:

@030 I can offer my experiences since asking the question, albeit it's a subjective view as I have no quantitative data. I can offer that a year and a half later, I haven't regretted my decision putting the master into us-west-2. Our users in any region don't seem to be impacted by having the master in us-west-2, at least from a UX perspective. I suppose at some point it could be worth measuring write times -- but to date I haven't had to. There's no business impact from any latencies due to the location of the master, that we can detect at least, so we're happy here


Blue Green Deployment DB transactions management

As per Martin Flowers' definition of Blue-Green deployment

  The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.

Steps

You've said that DBs are synced using master-slave topology.
So, to switch traffic from one instance to another, we have to make  steps:

Switch web-traffic from APP v1 to APP v2
Shutdown deprecated DB Blue
Promote DB Green to master mode
Start new DB Blue v2 in slave mode
(you may ask - why shutdown DB Blue instead of using master-master replication or other ways to sync DB Blue v1 with DB Green in master mode. See explanation in DB replication)

Graceful switch web-traffic

To switch web-traffic from APP v1 APP v2 - just use nginx reload - it will graceful end all working connection and move traffic from App v1 to App v2
There are many scripting solutions for switch blue/green using shell/python scripts

E.g.:

meappy/docker-nginx-blue-green: Docker blue green upstreams for nginx with ngx_http_perl_module
Simple 0-Downtime Blue Green Deployments: Python simple 0-Downtime Blue Green Deployments example

The logic is simple: add green upstream in nginx config, use it as default and send reload to nginx:

Sample nginx.conf for reverse proxy/load balancer

http {

upstream appv1 {
    zone appv1 64K;
    server 10.10.0.1:80;
}

upstream appv2 {
    zone appv2 64K;
    server 10.10.0.3:80;
}

server {
...
location / {
    proxy_pass http://$appv1;
}
}

}

Replace proxy_pass http://$appv1; with proxy_pass http://$appvv;

And then

nginx -s reload

Stop DB Blue

systemctl stop postgresql

Promote DB Green to master mode

Trigger the promotion using your defined trigger in $PGDATA/recovery.conf

touch $PGDATA/failover

Remove failover trigger from Green DB

cd $PRIMARY_DATA
rm -f recovery.* failover

Make sure that hot stanby mode is on on Green DB in master mode

cat postgresql.conf | grep '#hot_standby = on'

DB replication

Make a backup of master Green DB

psql -c "SELECT pg_start_backup('Streaming Replication', true)" postgresql://postgres@GreenDB/postgres

Send a backup to Blue DB

rsync -a $PG_DATA_Green_DB/ $BlueDB_IP:$PG_DATA_Blue_DB/ --exclude postmaster.pid --exclude postmaster.opts

See also

Evolution of Fault Tolerance in PostgreSQL

DB Migration (More complex but more robust database migration scheme)

You may use more complex scheme for migration using Ansible and `Pglupgrade tool:

See for details:

Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (Part I)
Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (Part II)

P.s some definitions.

Just to...

  ... sort the buyers from the spyers, the needy from the greedy...

And blue/green deployment vs canary deployment vs A/B test

Definitions, kinds of deployments and tests

(see details in these answers )

Blue-Green Deployment - When deploying a new version of an application, a second environment is created. Once the new environment is tested, it takes over from the old version. The old environment can then be turned off.
A/B Testing - Two versions of an application are running at the same time. A portion of requests go to each. Developers can then compare the versions.
Canary Release - A new version of a microservice is started along with the old versions. That new version can then take a portion of the requests and the team can test how this new version interacts with the overall system.
Feature flagging - The action of "configuring" (cold, or even hot) which functionality is (not)available for which (group) of users. If you also do something like "feature flagging" you can deploy first, measure soundness of your release in backwards compatibility/bug perspective, and release new functionality gradually to different users, or vice versa (scale down or even rollback functionality and/or binaries). Feature flagging allows for splitting availability of functionality from deployment of binaries, and gives much more fine-grained decision making then only "deploy/rollback"

P.P.S Blue/Green vs Canary

source

Both blue-green and canary releases solve the same purpose
Although both of these terms look quite close to each other, they have subtle differences. One put confidence in your functionality release and the other put confidence the way you release.

Blue-Green Deployment

  When deploying a new version of an application, a second environment is created. Once the new environment is tested, it takes over from the old version. The old environment can then be turned off.

1. It is more about the predictable release with zero downtime deployment.

2. Easy rollbacks in case of failure.

3. Completely automated deployment process

4. In cloud environment where it is easier to script & recreate infrastructure, blue/green deployment is preferred as it allows the infrastructure to be in sync with the automation

Canary Release

  A new version of a microservice is started along with the old versions. That new version can then take a portion of the requests and the team can test how this new version interacts with the overall system.

1. The canary release is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure.

2. It is about to get an idea of how new version will perform (integrate with other apps, CPU, memory, disk usage, etc).

See also illustration (source):

Blue/Green:

Canary:


This works exactly in the way you're describing.  You have two pods which each have a limit of 4 CPUs scheduled on the same node; so the total of the limits on that node is 8 CPUs; which is 200% of the node's physical capacity; and if the two nodes combined go over the node's physical abilities then the host kernel will time-slice available resources.  For CPU there's nothing especially wrong with this.

The exact same presentation in the kubectl describe node output applies to memory, and there is the potential for trouble with that: if your pod limits add up to 200% of a node's physical memory, and together they exceed the physical capacity, then the host kernel will respond by killing one of them off.  Lower requests will allow more things to be potentially scheduled on the same node, but keeping requests closer to limits gives you a better chance of them staying running long-term.


Releasing so many changes at once to production

This question as it is posed is quite leading - the use of adjective "bad" to describe what you are doing implies that there is a "good" way. Nobody can answer whether your process is "good" or "bad" but you, and it should be judged on the effectiveness of the process. Perhaps this way of doing things is good for your environment...
However, what we can say is that the situation you describe goes against the principles of continuous delivery and continuous integration.
For example, describing the principle of "work in small batches",
the Continous Delivery website states:

In traditional phased approaches to software development, handoffs from dev to test or test to IT operations consist of whole releases: months worth of work by teams consisting of tens or hundreds of people.

This sounds like your situation.
It goes on to state:

In continuous delivery, we take the opposite approach, and try and get every change in version control as far towards release as we can, getting comprehensive feedback as rapidly as possible.
Working in small batches has many benefits. It reduces the time it takes to get feedback on our work, makes it easier to triage and remediate problems, increases efficiency and motivation, and prevents us from succumbing to the sunk cost fallacy.

So, by this comparision, yes, your release management is "bad", because it doesn't work in small batches, or continuously. (Again, with the caveat that any process that is executable effectively is a good process).
Another widely-used reference text, the Google Site Reliability Engineering Book has a whole chapter on release engineering. It says:

We have embraced the philosophy that frequent releases result in fewer changes between versions

Your process seems to run counter to this statement, and though you don't specify what level if testing or canary deployments you use, one may surmise from the fact that there is no continuous integration that this is very small, if any. Therefore, by "SRE logic", every new deploy is high-risk, because it contains a large amount of untested changes.
There is another aspect which increases the risk in doing things "your" way. It is not reasonable to expect that a CAB will be able to make an informed decision on whether or not a given build should be released, without feedback on the performance or behaviour of the application in it's target environment (i.e. prod). They will be making decisions based on predictions and assumptions, rather than data.
For all these reasons, and probably many more, I think the consensus of this community would be that the answer to your question would be "yes, this is a bad way of doing release engineering".
A better way would be to:

Have a minimum of testing in place, and work on extending that with every deploy
Start to automate the tests as far as possible and have clear documentation for how to test it when automation is currently infeasible.
delivery artifacts continuously and run canary deploys to a test environment; work on aligning  that environment as closely as possible with the production environment
Aim for the option of deploying any given build, and have a means for rolling back in the case of problems, to reduce the fear associated with deployment.

Smaller, and more deployments mean smaller changes at a time, which are easier to fix, and observing the deployment procedure continuously will give better insight into its reliability and where to improve things.
Automate, measure, improve. Repeat.


Jenkins "internal" pipeline script producing errors when running sh steps

Apparently this was caused by the pipeline name having an apostrophe character in it. Renaming the build solved the issue.


How to enable Google Maps and generate an API key using Terraform?

I faced the similar problem recently, but fixed it by providing additional roles to the service account. I am not sure what permissions you need, you need to look it up. In my case, I was trying to create an appengine project, So I had to add "Appengine Admin" role to the service account and enable "appengine.googleapis.com" via terraform resource "google_project_service"


Tool for storing per environment configuration

There are many tools that can do something like this, including configuration management tools like Chef, Ansible, or Puppet; and KVS tools like Consul and etcd. You could also integrate it as a build step in your CI server, or sidestep the issue using live configuration at runtime against an external configuration store (again, something like Consul or etcd, or any database at all).


How to automatically remove k8s namespaces every week?

I recommend that you have an expire-date as label or annotation on the namespaces used for test. Then you can have a Job to clean-up expired namespaces.


Is there a way to set build parameters outside of the pipeline directive?

EDIT:
It might have a chance for me to misread your question at the first time, so let me add another one.

If you meant by Jenkinsfile as you have a manual parameter input in Jenkinsfile for Multibranch or Github Org pipeline, then probably bad idea.

Those kind of jobs should run with predefined parameters or take them on-the-fly like manipulating some environment variables without user input.

You can simply make another pipeline job for manual trigger if the predefined parameters can't cover all the case you need.

And for run function, it looks like build job is what you are looking for.

https://jenkins.io/doc/pipeline/steps/pipeline-build-step/#-build-%20build%20a%20job

build job: 'Another job in same Jenkins server', parameters: [string(name: 'param1', value: ${new_value})]

--------------------------------------

Are you sure the parameter input is waiting for slave containers?
or do you use some special choice plugins for build parameter?

I think a job is even not starting until a user fills out the parameters and hit the build button in Jenkins UI - so no container required until then.
We had same k8s slaves, but never had that issue.

(But some parameter plugins might need some pre process..)


Update ECS task definition with new image from ECR

Options are:

Use CodeBuild to push the new image. Make sure the task definition is using the "latest" tag. You'll need to force the deployment of the task definition to pick the new image.
Use CodePipeline to update ECS. It will automatically generate a new task definition revision with the new image and deploy it.


How to create a MariaDB/MySQL image using utf8 instead of the default latin1 charset?

In the official MariaDB image, my.cnf is configured to read additional configuration files from the /etc/mysql/conf.d/ and /etc/mysql/maria.conf.d/ directories:

root@bbd24e22be98:/# tail /etc/mysql/my.cnf

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!include /etc/mysql/mariadb.cnf
!includedir /etc/mysql/conf.d/

root@bbd24e22be98:/# tail /etc/mysql/mariadb.cnf
# * Character sets
#
# Default is Latin1, if you need UTF-8 set all this (also in client section)
#
#character_set_server   = utf8
#collation_server       = utf8_general_ci
# Import all .cnf files from configuration directory
!includedir /etc/mysql/mariadb.conf.d/

Based on that, it is quite easy to setup utf8 everywhere by writing a configuration file to override the relevant settings, copying it at the right place from your Dockerfile:

$ cat db/utf8.cnf
[client]
default-character-set=utf8

[mysql]
default-character-set=utf8

[mysqld]
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake

$ cat db/Dockerfile
# Base image
FROM mariadb:10.4

# utf8 everywhere
COPY utf8.cnf /etc/mysql/mariadb.conf.d/
ENV MYSQL_RANDOM_ROOT_PASSWORD=yes

I let you compare the output of show variables after those changes with the one provided in the question:

MariaDB [(none)]> show variables like 'char%'
    -> ;
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+


How to determine which packaged pip to install in order to run the Ansible pip module?

There are two options either you make sure you use the same Python version on each host or you make sure the right packages are installed on each platform.

Chose same Python interpreter

Provide the python binary to use. Since you have a Debian on each host, the location is always the same. If you know a Python version that is available on each host, go for this.

Either in your inventory hosts

[webserver]
srv1.example.com
srv2.example.com

[webserver:vars]
ansible_python_interpreter=/usr/bin/python3

Or in your Ansible configuration ansible.cfg

ansible_python_interpreter=/usr/bin/python3

See also https://docs.ansible.com/ansible/latest/reference_appendices/python_3_support.html#using-python-3-on-the-managed-machines-with-commands-and-playbooks

Package per Python version

If you do not know whether either Python 3 or 2 is available on each host, ensure to get the right package for each Python version.

- apt:
    name: python-pip
  when: ansible_python.version.major==2

- apt:
    name: python3-pip
  when: ansible_python.version.major==3

You may face the same issue when installing Python packages. So you can put your Pythone version specific tasks in a v2 and in a v3 file and conditionally include one for version 2 and the other for version 3.


kubectl port-forward forwards connections to a local port to a port on
  a pod. Compared to kubectl proxy, kubectl port-forward is more generic
  as it can forward TCP traffic while kubectl proxy can only forward
  HTTP traffic. Generally speaking, using port forwarding you could get
  on your ‘localhost’ any services launched in your cluster.

https://stackoverflow.com/questions/58360323/difference-between-kubectl-port-forwarding-and-proxy
It does not make a lot of sence to run 'kubectl proxy' from inside VM running k8s cluster. Kubectl is your client tool, so stick with it outside of cluster range.
The way you would access production services, for the most part if your service is DB oriented stick with ClusterIP. If you want to externally access some Web/HTTP service expose it as a Load Balancer or an Ingress ( in case u have N of them ). If you want to test the DB you could just run a dummy pod and execute some sql-client commands from there.


Error when using 'ansible-local' provisioner with Docker container

It seems that there is some bug in Hashicorps Packer 1.2.0, consider downgrading or upgrading to v1.2.1, comments in GH issue page confirm that the deployed fix is working.


Why do I need tools like minikube or microk8s to run k8s locally?

Kubernetes has several system-level parts.  It needs a functioning etcd cluster, the API server proper, and a per-node service called the kubelet.  It also generally expects to be able to use 100% of the resources available on any given node for its own use.  Just having the kubectl binary on its own isn't enough to use Kubernetes; you also need the rest of the cluster that goes with it.

There's no specific reason you can't directly install all of the Kubernetes cluster parts on the host (using kubeadm, or Minikube's "none" VM driver), but it will be trickier to maintain.  It may or may not conflict with local Docker use, and it will be tricky to uninstall or upgrade it.  If you run Minikube in a VM (or Kind in a Docker container) then all of the parts are preinstalled for you, and you can easily stop or delete the VM (or container) when you don't need it any more.


Terraform: ignore_changes conditionally? (temporary EC2 instance public IP address)

Maybe you could use "count" attribute for this?

resource "cloudflare_record" "record1" {
    count = ${aws_instance.instance1.public_ip == "" ? 0 : 1}"
    ...
}


Easiest way to manage an elasticsearch cluster in GCP?

One could use helm to install elastic on managed k8s in GCP.


Can the Waterfall method leverage CI/CD+ and still be considered Waterfall?

I don't see why not.
Sequential SDLC models, like waterfall, are about the organization of steps. The requirements are "finalized" before extensive design begins. The design is "finalized" before the implementation begins. System testing happens after the requirements are satisfied. Some waterfall-like models, such as the sashimi model with some overlap between these phases or waterfall-with-subprojects, are multiple waterfalls after the overall system requirements and architecture have been completed.
There's nothing in the organization of these steps that would prevent continuous integration or configuration of code. Continuous deployment to a development environment is also possible, but continuous delivery or continuous deployment to production may or may not be possible, depending on the usability of interim builds before the requirements are complete. These practices can still be used once system implementation begins. CI and automated tests can help developers find issues during development before system integration and system testing activities begin. Configuration as Code can help with the configuration management of the system, allowing it to be versioned alongside the software system and rolled out to different test and production environments when it's time.
The key difference between highly sequential models and iterative and incremental models is responding to feedback, not the practices that can be used. Some practices may not make as much sense to use, especially if they are about getting and responding to feedback. If you have "finalized" requirements and "finalized" designs before you can start getting feedback on the system, stakeholder feedback will likely change those requirements and/or designs. Because of the scope of the work of requirements and design and the impact on the built system, late changes are expensive.


What are benefits of using third party app metrics tools versus custom build tools?

A third party app metrics tools is great if it already covers all (or most of) the parameters you want to monitor and it presents the information in a convenient enough manner for your use case - you just need to (find it first and then) learn how to install/use/maintain it and then do that.

Advantages:

avoids the software development and maintenance costs, which can be quite high depending on the capabilities
can be immediately used, a custom tool needs to be developed first
technical support (not always great, tho)
standardisation: if the tool is a popular one - expertise on it can be requested on a job description instead of having to train every user on it

However there are many cases in which custom development is needed, some already mentioned by Michael: reliability, security/confidentiality, lack of integrations for your custom applications.

Depending on each particular case, custom development can go from tiny scripts extending the coverage of a third party tool to complex wrappers assembling multiple such tools to achieve the necessary monitoring capabilities

Such development is not always lightweight:

it may require in-depth tool knowledge and/or language expertise outside of your team
it may need management for heavy/complex or even conflicting dependencies
the adaptation layers for the tool's actual usage environment may be very heavy

Sometimes the cost of such development is higher overall than that of developing a custom system.

Othertimes the functionality actually needed from the third party tools is a relatively small percent of its capability set and the overhead for learning/installing/operating the tool is simply not worthy.

You guessed it - I wrote my own monitoring tool :)


How can I issue a Let's Encrypt certificate with Ansible when 'challenge_data' cannot be found?

This was caused by the fact that I was starting the Let's Encrypt challenges using a with_items loop. From here:

  When using register with a loop, the data structure placed in the variable during the loop will contain a results attribute, that is a list of all responses from the module. For a more in-depth example of how this works, see the Loops section on using register with a loop.

The solution is to have a separate task for each Let's Encrypt challenge, rather than having them all in one.


Does "docker run" actually SSH into the container?

No, it uses an internal "attach protocol" via the Docker daemon.


Issue connecting to ECR using Spinnaker on Kubernetes

Update for anyone else who runs into this issue, when Spinnaker is installed using helm it's using an older version of the Halyard container which doesn't have the AWS CLI tool. This seems to be hard to resolve as just updating the values file for helm to the latest version doesn't seem to work and I didn't dig much into the specifics as didn't want to go down that rabbit hole.


Fabric (and Capistrano, presumably the unnamed Ruby tool you came across) are a bit unusual in that they're task runners with extra features for easily running tasks on remote hosts.  I'm not aware of any other popular tools that do exactly the same thing, but depending on your needs there are a few other options that may work for you.

Grunt and Gulp are the foremost task runners in the Javascript ecosystem.  While not designed specifically to deploy software, they both have plugin-based systems that allow you do this, and you'll find plenty of articles of people using either one to do so.  For instance, this tutorial uses the grunt-ssh plugin to integrate ssh with grunt, while this one relies on gulp-rsync.

Using task runners to deploy code tends to be an approach favored by developers who are wearing an ops hat.  When you ask an operations engineer, they're more likely to suggest hacking this behavior into a configuration tool.  This can take many forms (for instance, packaging your app into an rpm or deb, then installing it like any other software), but I'm going to focus on what I think would be the simplest and most direct method for you.

Ansible is ostensibly a configuration management tool, but it really shines at task orchestration, which I would define as "running a set of actions across a set of servers".  The official documentation is a good place to get started, but to give you an idea of what this might look like, you might have a playbook (a YAML file that defines a set of tasks) like this:

- hosts: webservers
  tasks:
    - name: update code checkout
      git:
        repo: ssh://git@github.com/mylogin/hello.git
        dest: /var/www
        version: master

    - name: restart webserver
      command: restart-command-you-run

When you run this file with ansible-playbook, it will look up your host list (as simple as an ini file, but can also be a dynamic list written in any programming language) to determine what servers exist and what groups they're in, and then run the defined tasks on the servers that you tell it to.

Now, you specified that you wanted to stay in the Javascript ecosystem.  However, while Ansible is a new thing for you, it doesn't have all the complexities of another programming language, since configurations are written in YAML.  It will also allow you to do much much more later, and unlike several of its competitors, uses ssh to communicate, so there's very little setup involved (nothing to install on the remote machines).


Sure. You can run a Linux-based Docker image on any Linux host.
Yes, it is possible. You can run & develop in a debian-based docker container.


How to define dynamic parallel stages in a Jenkinsfile?

This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:

We have a shared library where we keep our scripts ins ./vars/someScriptName.groovy
We created a groovy script that generates the stages.

Here's a working pipeline code for you to try it out:

def generateITParallelStages(body)
{
    def config = [:]
    config.stages = []

    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

    def buildStages = [:]
    config.stages.each { String stage, closure ->
        // default settings, if you want to have any
        def settings = [:]
        settings.type = 'online'; // if you don't define the type, it will be 'online' by default
        settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default

        closure.resolveStrategy = Closure.DELEGATE_FIRST
        closure.delegate = settings
        closure()

        buildStages.put(stage, prepareStage(stage, settings))
    }

    return buildStages
}

def prepareStage(String stage, Map settings)
{
    return {
        print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
    }
}

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                script {
                    parallel generateITParallelStages {
                        stages = [
                                webgoat: {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'juice-shop': {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'our-website': {
                                    type = 'online'
                                    suite = 'internal'
                                }
                        ]
                    }
                }
            }
        }
    }
}

And here's the result:

This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this).

Other people seem to be having a hard time figuring this out as well - JENKINS-53032. If you feel the same make some noise :)


Problem solved:
It's because of endianess of output.
There is an option --endian which by default is little.
The example string was cá, which translates to 63 C3 A1 utf-8 encoded:
~ $ echo -n cá | od --endian=big -x
0000000 63c3 a100
0000003

(The -n prevents \n in the end of string, making last byte 00)
So it's exactly the same output, but with bytes inverted:
~ $ echo -n cá | od --endian=little -x
0000000 c363 00a1
0000003

(c363 instead of 63c3 and so forth).


Is there a way to map ports two docker container

On Linux, containers live in separate network namespaces, so localhost or 127.0.0.1 on the container is not localhost on the host. I'm not sure about other OSes. You could expose a port to the host like this:

docker run -p 3306:3306 mysql

Then you can access it from any container at the host IP address, which for the default network is 172.17.0.1:3306. You can add -p many times, read the docs here: https://docs.docker.com/

All that the -p does on Linux, is essentially:

sudo iptables -t nat -A DOCKER \! -i docker0 -p tcp --dport 3306 -j DNAT --to 172.17.0.2:3306

(Or whatever your mysql container IP address is.) Docker compose and similar utilities make containers able to see each other with --link eg.

docker run --link web --link mysql:localmysql app

The only thing this does is to put the correct IP address for the container in a file called /etc/hosts inside the container (but this might change in future.)

If you want to use --link, all containers need to be on the same network. You can connect them while running with:

docker network connect mynet web

The main reason for doing this is because the containers don't have fixed IP addresses. You can do:

docker run --network mynet --ip 172.18.0.10 -d app

To give your container a static IP - and then just use that instead of the name. The docker team keeps changing how things work, so you might need to run additional commands to allow different networks to connect to each other.

You can of course also do all of this manually, on Linux, using the firewall, on the host machine. (Start learning about it here: https://www.google.com/search?q=iptables (Also look at the Images tab!)

iptables -nvxL # shows the filter table
iptables -nvxL -t nat # shows the NAT table where port forwarding is done

To dynamically forward a port to a container, get its IP address, eg 172.18.0.10, then:

iptables -t nat -A POSTROUTING -p tcp --dport 1234 -j DNAT --to 172.18.0.10:4567

Be keenly aware that if you do this, and your container restarts, or gets another IP address, that this will not automatically update, and unless you use iptables-save and iptables-restore in your startup and cron scripts, these manually created rules will disappear. Also that the docker command and compose scripts won't know about any configuration that you manually add like this.

If you really want localhost in your container to point somewhere else, you can edit /etc/hosts inside the container, and change the IP address of localhost - be very keenly aware that other things might break, because localhost is supposed to point to the same host, not another one, so there will be unintended consequences at some point.


Azure Devops Pipelines Triggers not working

If you have set a CI trigger make sure you have set the filter to the exact branch which you need the build to be triggered and not the entire repository which is the default case.


When I raise build size issue as a concern, it usually doesn't come from "it is so big, it will be expensive to store it".

The main problems with large builds are the following -

increased shipping time. moving big bits from place to place frequently is time consuming.
frequent changes to big artifacts plus a large enoug retention period make storing such artifacts become costly, less so with layered artifacts like docker images.
creating such big artifacts usually is time consuming, more so than creating much smaller artifacts. automating the process to create smaller artifacts might take time, but the automation that is repeatable should be as short as possible to allow for fast feedback.
recovering from failure (depending on configuration) might take more time with larger artifacts, especially when an older artifact needs to be reapplied instead of a faulty new-er one.

I subscribe to the four devops metrics:

Lead time for changes - shorten it
Deployment frequency - increase the frequency
Time to restore service - shorten it
Change failure rate - reduce it to never

Large artifacts usually create a problem in each of these metrics, and none of these metrics are really concerned with the cost of storage really - because that is cheap, time is expensive.


How to get 'diff' on Windows TravisCI machines?

When git is installed it usually doesn't add the diff executable to the path because it would conflict with other versions of diff native to Windows, for example, the Compare-Object alias in PowerShell.

You can usually get around this by adding C:\Program Files\Git\usr\ to the path which contains all of the UNIX ports of functionality such diff, grep, awk, sed.  You do have to be careful as some commands are used more commonly on Windows than others, the most common one is find.
A second option would be to fully qualify the call to diff by appending C:\Program Files\Git\usr\bin.


The key difference between DevOps in the context of Avionics and, aircraft/spacecraft merely has to do with deployment velocity due to FAA certifications, Aircraft re-grooms and the way in which the Airline industry releases their changes.

But ultimately, DevOps is not exclusive to High Velocity delivery methods (like Agile). And can be done in the context of Waterfall methods or other similar methods.

In some ways, the airline industry is often factory oriented, which aligns well with Kim, Behr, and Spaford's depiction and conceptualization of DevOps.

Development on LRUs and racks can easily be done using an Agile methodology using a DevOps mentality. These many small changes can be stored up and released periodically to undergo final rack testing and certifications (such as AS9100). Airlines could be doing much more to leverage automation for software deployment to fleets and have yet to capitalize on these developments.

At the Avionics company I worked for for several years, we were seeing several successes and challenges with implementing DevOps. I simply can't emphasize enough however that challenges were internal and political, not technical or methodological challenges. These are also not systematic issues resulting from industry regulations, policies and standards. Avionics companies who are able to innovate and provide better value, higher reliability and cheaper costs will easily be able to out-innovate their competitors and gain traction and market share. The Avionics industry is ripe for this kind of shakeup and DevOps is well suited to be a cause of changes.


How to protect secrets whilst enabling the ability to amend a pipeline

Developers or even admins of your GitHub repository wouldn't be able to see secrets configured there by default. Even if you debug GitHub Actions workflows and accidentally log secrets, GitHub would mark them with ***.

So the only attack vector I can think of is highly deliberate and it would require modifying GitHub Actions workflow code and egressing the secret outside to some pre-configured recipient on the Internet.

If you're worried about such attack vector beyond regular code reviews, you may try exploring CODEOWNERS as a way to protect .github directory - https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners.


In my experience, the easiest way to do this is to put Jenkins behind an SSL-terminating proxy, such as Apache or Nginx.  You can also configure SSL within Jenkins itself, but I've found that using a proxy is usually a lot easier to set up and manage with regards to SSL.


What is the most used tool for Cloud Infrastructure deployment Ansible vs Puppet vs Terraform?

@jasper welcome!

This really might depend on the context. On long term it's worth to take a deeper look for a moment if you are not after quick shots and go beyond experiments for a sustainable (and large) infrastructure.

So we have got many clouds out there. Deployment automation means, they have some API.

So what does make a deployment automation tool delivering infrastructure as code capability?

(1) Unified, programmable, configurable abstraction layer, also called domain specific language (DSL).
(2) An object model of (multiple) cloud assets and available services.
(3) Corresponding logic to execute actions on the assets typically: create, read, update, delete (CRUD) and synonyms.

Now there are some challenges to address human readability and sustainability varying by programming language ecosystems and communities:

Cloud products being themselves computer infrastructure and network engineering (these are different disciplines!) abstractions are not following some official "cloud domain" ISO/IEEE naming standards as far as I know. That is, all the names are pushed by vendors and might be marketing biased. So either you invent your own terminology or decide which terms are most generic, or a mix of that. There is therefore a good reason that you find also cloud vendor specific deployment languages and tools for example like Cloud Formation by AWS.
Somebody needs to maintain related components to support new API versions for every cloud platform!

Some deployment tools delegate the cloud driver (that's another term for cloud platform specific code bases) to separate projects, which in their term delegate responsibility for API compatibility to one or another boilerplate (=often boring and tedious) API implementations.

Therefore my conclusion is that depending on your purposes the most straight-forward way to write infrastructure as a code is the programming language of your choice using a well-supported SDK which is though vendor-specific!

Still this is the point where you have got greatest flexibility because for a small standard use case you could write your own abstraction layer and your custom logic whatever you want.

Python examples:

boto3 for AWS
exoscale/cs for Apache CloudStack
python-openstackclient for Apache CloudStack

Recent Terraform example: its Apache CloudStack driver has backward compatibility problems because it's based on some less-visible CloudStack API SDK written in Golang. I am sure you can run into problems of this sort with any deployment tool; so if you commit to one, bring enough courage and patience to connect with and contribute to the corresponding community (same is for SDKs but they at least typically have less abstraction).

That is, before deciding for a certain cloud deployment automation tool, consider besides the things you see on the surface like syntactic sugar and community support also the following aspects:

If it's open source, is it written in a programming language where you/your team could create an emergency patch if things go really worse?
How well supported are those cloud drivers critical to your business: these typically can't be all of them, if so, you might run into issues like GitLab with Docker Machine (also a specific deployment tool after all), where a significant technical base for a competitive product feature suddenly breaks away because all the responsibility in this context had been factually outsourced to the open source community.


Is there an alternative to setting up the whole project workspace in a Jenkinsfile

Using Docker with Jenkins in a situation like this is a great way to

Keep Jenkins worker slave nodes clean (so you don't have to build or install anything on them)
Have all dependencies and environment in an easily manageable artifact (in this case, a docker image is the snapshot of the exact OS, filesystem, env and dependencies)
Vendor / keep copies of dependencies at a certain point in time (Example: You don't want to update at the same rate as the dependency)

If you have some kind of artifact management system (Example: Nexus) you can store your docker images in the artifact management system, and then pull them into Jenkins, and create a running container that has your repository in it. From there you can build project run tests inside the docker image. Just be sure you remember to move/copy all test results and built artifacts out of the docker container if you need them or use them later.

If you need more details or specifics please edit your question and I will edit my answer accordingly.


Docker managers are losing quorum on AWS

It seems like the AutoScaling Group from AWS are not that synchronized with the Docker Swarm configuration and the default timeouts in the AutoScaling are not accepted in the Docker Swarm managers.

Actually the Health Check is not configured good by default in the EE CloudFormation template, even if the managers are running and stable the Health Check fails and the AutoScalling Group tries to launch new managers. New launched managers are not able to become cluster participants, right after their running state.


Docker security risks passing secrets over environment variables

ENV variables are a good way of passing secrets to a container. What you don't want to do is bake the secrets directly in the image. As you have rightly done, do not store secrets in your source control repo as well.

Besides ENV variables, you could also consider using an external secrets repository, such as Hashicorp Vault, or possibly even AWS secrets. See also vault's github page. Your container would still need a token/credentials required to Authenticate/Authorize with the key store, but if required, these can be made to expire/rotate.

So you are on the right path. Some additional reading is worthwhile at this point:

https://medium.com/@mccode/dont-embed-configuration-or-secrets-in-docker-images-7b2e0f916fdd
https://techbeacon.com/how-keep-your-container-secrets-secure


There is Tomcat configuration file for the logging: TOMCAT_HOME/conf/logging.properties.
Edit the file and remove all the handlers except the console handler:

.handlers = java.util.logging.ConsoleHandler

This will direct all logging to the console, except for access logs (localhost_access_log.XXXX.txt). It is always written to file by the AccessLogValve, configured in 'server.xml'. To redirect that one you need to configure own implementation of the AbstractAccessLogValve... which requires coding.


The key to presenting solutions is always to identify problems first; if the team doesn't feel like you're solving any problems, then they're going to dismiss what you're suggesting.  The specific problems your Ops team is facing we don't know, but there are a number of common ones:

Getting non-actionable alerts
Spending lots of time in toil
Version upgrades are a pain
Deployments are a pain
No visibility into what the apps are doing

and so on.  Once you know what problems they're having, then you can target solutions that fix those.

  The main subjective problem seems to be that they think real bad about developers.

Cultural changes are incredibly difficult to make (usually the easiest way to is hire people with the philosophies you want and gradually prune off the old people).  But embedding operations engineers with developers can help this one immensely.  I've worked in several small companies where there was just one engineering team (and I happened to work primarily on ops work), and so this idea comes very naturally to me, but for people who have been working solely on an operations team with little contact with developers for a long time, it can be a difficult adjustment.  But move your desk to be with them, participate in their chat channels, eat lunch with them - this all helps both ops and devs realize that the other side are actually humans.  This also provides better opportunities for ops to use their expertise to solve problems for devs that otherwise wouldn't've surfaced (say, they're trying to do something and you provide a quick shell script that solves it much faster than the manual way they were going with), and that builds trust and emotional budget that you can spend when you need something from them.


How do I setup ssh from jenkinsci/blueocean docker to remote?

You can run an ssh server in the container as the ENTRYPOINT. It can either be started in the foreground :

ENTRYPOINT ["/usr/sbin/sshd", "-D","-p","22"]

Ensure that you have openssh installed and host keys setup.

Perhaps this Ansible role would be illustrative


Purchased AWS reserved instance in wrong region

If you have done payment of your reserved instance then you are unlucky. now you have only one option Sell your reserved instance in AWS marketplace.

But If you till not done payment then you lucky, I suggest don't do payment till next billing cycle. AWS by default cancel your reserved instance if you don't do payment after the end of the month.


What is the utility of an inhouse DevOps if cloud providers already provide professional services?

Some good reasons could be:

The salary of an inhouse devops is pretty sure less expensive as the consultants.
The consultants are only assigned during a limited period, to temporary add (wo)manpower, e.g. while putting together the infrastructure.
The consultants come with specific expertise that the inhouse devops do not (yet) have, whereas part of the job of the consultants is to perform knowledge transfer and/or on the job training for the inhouse devops.
The consultants are there to do the boring parts of the work, whereas the inhouse devops are assigned to the strategic / long term tasks (so that when the consultants leave, the know how doesn’t leave the company also.

BTW, IMO the best consultants are those who:

Say their jobdescription is like “My job is to make sure that you don’t need me anymore ...”.
Never touch the keyboard, and instead always coach / help inhouse devops to have them complete any task (it seems so easy when you see somebody else complete some task, but it is way more difficult to complete it yourself).


Is Azure WebDeploy safe for production Web Apps?

WebDeploy by itself cannot guarantee if your application pool will not be recycled during deployment. To be on the safe side, I would use deployment slots.
Also I would recommend to tag question with azure-web-sites so right people will help you.


WinRM / WSMan - getting SSL handshake error / TCP RST when connecting with IP address, correct connection when connecting with hostname

I faced the same error .. You probably want to run ConfigureRemotingForAnsible.ps1 script with -ForceNewSSLCert option on your remote machines.

.\ConfigureRemotingForAnsible.ps1 -ForceNewSSLCert


Which ways are there to decouple deployment from release?

For software in the web app category, depending on you infra/hosting provider such decoupling might be possible to switch incoming traffic across (or split it between) different deployed versions of the sw, practically covering any of the changes you mention: bugfixes, visuals, etc.

Such support would typically not require feature toggles. And it might be applicable regardless of the app being monolithic or split in microservices.

For example Google's App Engine Paas offering has support for traffic splitting and migration.

From Splitting Traffic:

  You can use traffic splitting to specify a percentage distribution of
  traffic across two or more of the versions within a service. Splitting
  traffic allows you to conduct A/B testing between your versions
  and provides control over the pace when rolling out features.

From Migrating Traffic:

  Traffic migration switches the request routing between the versions
  within a service of your application, moving traffic from one or more
  versions to a single new version.


Gitlab CI - Deploy to an active docker container

The image acts as a foundation for each container that you start from it.
You might do the update in your 'fashion' way, but containers are ephemeral - let's say it got restarted, you updates are lost and you are back to the previous version.
If you want to handle this process by yourself, how would you track your versions ? Essentially you would have a root docker image and after that it would be so hard to know which version is currently running ? Could you also safely roll-back version to version ? How about if somebody hijacks you container, due to some Unix Socket permissions with the Docker daemon ...
What I am trying to say, do not rediscover the wheel :)
Take advantage of modern technologies like Kubernetes, deploy each image for a new version there, you have a roll - out deployment, readiness probes and so on ... You would be safe that your software is always working, track the versions and have H/A.


simulate post step for dynamically generated Jenkins pipeline stages

You can't use post because post is part of the declarative DSL, not scripted, and you're defining your stages in a script block.  So I'm afraid that as long as you're using a script block to define your stages, you're going to have to stick with the scripted method, which is the groovy try { ... } catch { ... } finally { ... } construct.  For example:

def createStage(String stageName) {
    stage(stageName) {
        try {
            echo "Stage: ${stageName}"
            echo "${stageName} > this is executed only on success"
        finally {
            echo "${stageName} > this is always executed"
        }
    }
}


Is it possible to have pair of IPs in Ansible?

Q: Is it possible to have a pair of IPs in Ansible?

A: Yes. It is possible. For example, it might be useful to create groups in the inventory with all pieces of information.

[primary]
1.1.1.1
1.1.1.3
1.1.1.5

[standby]
1.1.1.2
1.1.1.4
1.1.1.6

[ha]
1.1.1.1 stdby=1.1.1.2
1.1.1.3 stdby=1.1.1.4
1.1.1.5 stdby=1.1.1.6

It's possible to simplify the fetching of the configuration.

- hosts: primary, standby
  remote_user: root
  tasks:
  - name: copy the config to local
    fetch:
      src: /data1/cronjobs/gen_ddl
      dest: /AnsibleDir/conf

Without flat: yes the configuration files will be stored in

/AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_ddl

Then it's possible to reference the configuration files by the inventory hostname. For example groups['primary'] (secondary the same way). Take a look at myshell_output and select the data for content and dest from the items

- hosts: 127.0.0.1
  connection: local
  remote_user: root
  tasks:
  - name: loop primary config
    command: "/AnsibleDir/driver.py
              /AnsibleDir/conf/{{ item }}/data1/cronjobs/gen_ddl"
    register: myshell_output
    loop: "{{ groups['primary'] }}"
  - name: copy the output to a local file
    copy:
      content: "{{ <SELECT-FROM-ITEM> }}"
      dest: "/AnsibleDir/output.{{ <SELECT-FROM-ITEM> }}"
    loop: "{{ myshell_output.results }}"

To reference primary/standby pairs use group ha. For example the play below changes the configuration both at primary and standby for all hosts in ha.

- hosts: ha
  tasks:
    - name: Change configuration at primary
      lineinfile:
        path: /AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_dd
        regexp: <ADD-REGEXP>
        line: <ADD-LINE>
    - name: Change configuration at standby
      lineinfile:
        path: /AnsibleDir/conf/{{ hostvars[inventory_hostname].stdby }}/data1/cronjobs/gen_dd
        regexp: <ADD-REGEXP>
        line: <ADD-LINE>

  Q: I need to find the differences of configuration files in 2 machines. they need to be the same. In python script, it will find the differences and fix them. So I need to have both configurations on my local then run the python script for them.

A: Fix the configuration files in 2 machines

- hosts: ha
  tasks:
    - name: Fix primary and standby config
      command: "/AnsibleDir/driver.py
                /AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_ddl
                /AnsibleDir/conf/{{ hostvars[inventory_hostname].stdby }}/data1/cronjobs/gen_ddl"


Jenkins Exception that is caused by stdout redirection using windows cmd

I found out how to do it without pipeline. It only has to do with command line script:

start "" C:\MyApp\MyApp.exe >> text.txt

This call is asynchronous (i.e., the command line won't wait for MyApp.exe to return). If you want it to wait, you can use the /w flag, like this:

start /w "" C:\MyApp\MyApp.exe >> text.txt


What is the term for a deployment designed to make a future cutover seamless?

... by making a minor change in the v1 front half, this breakage can be avoided.

The above excerpt of this queston reminds me of something similar from around the late 90s, say 1996-1999 (when zillions of ITers were preparing for Y2K ...). In those days, new software releases of huge software vendors would come with release notes that included things like either of these:

"this release is to make this software Y2K tolerant " (= allows the software to contnue to work as of Jan 1, 2000, at least for a while, though it would keep showing dates with 2 digits for the year, like 00, and not yet 4 digits, like 2000).
"this release is to make this software Y2K ready " (= allows the software to continue to work as of Jan 1, 2000, without any (known) limitation because of the change of century from 19 to 20).

Y2K tolerant softwares would typically also include some sort of magical cut-off value, e.g.: years from 70 (when even MS DOS did not exist ...) to 99 were assumed to be from 1970 to 1999, while anything from 00 to 69 were assumed to be from 2000 to 2069. Such assumption was also why such softwares were required to be superseeded by some Y2K compliant release later on (in which those assumptions were obsoleted).

With the above in mind, the term you are asking for could be something like a "V2 tolerant release", which at some point in the future would then be followed by a "V2 ready release" (= your actual V2 release).


Using Jenkins gitlab token to make an API call

Try something similar to the following:

environment {
  GITLAB_API_TOKEN = credentials('gitlab_api_token').
}

or

withCredentials([[
  $class: 'com.dabsquared.gitlabjenkins.connection.GitLabApiTokenImpl',
  credentialsId: 'gitlab_api_token',
  variable: 'GITLAB_API_TOKEN'
]])

See https://jenkins.io/doc/pipeline/steps/credentials-binding/
and  https://github.com/jenkinsci/gitlab-plugin/issues/536 for more information.


How to get exposed ports from inside of container?

TL;DR: You don't.

From inside of the container, you do not know which ports are exposed since exposing a port is only documentation visible in the image and container metadata. To detect which ports are published, you'd need to know the host IP, which can be difficult to lookup depending on your environment, and then you'd need to do something like a port scan of the host and tcpdump inside the container to detect the incoming request.

All of this is non-trivial because it's solving the problem in the wrong direction. You build your app inside the container to listen on a specific port, configure that default port and potentially let users override it with a variable or config file. Then you document this decision be exposing the port in the Dockerfile with the EXPOSE command. This is only documentation, it goes in the image metadata, users and programs can see it, but by default docker will not do anything with it. Finally, users of your image will publish the port on the host when and where appropriate, and map to the port of your application.

In other words, the right side of the published port (the container port) is largely controlled by the image creator, not the user running the container. If they publish to the wrong port, it is not be the applications responsibility inside the container to dynamically adjust what port it is listening on, and this would be difficult to impossible to do if you had multiple ports anyway.


Bitbucket Webhook plugin for Jenkins - differentiate between branches

I believe it is now possible to configure the webhook on per-repo basis:

https://mohamicorp.atlassian.net/wiki/spaces/DOC/pages/381419546/Configuring+Webhook+To+Jenkins+for+Bitbucket+Bitbucket+Branch+Source+Plugin


I'd go with node.run_state to store a transient variable in a run and define it in a ruby_block so it happens at converge time, something like this:

yum_package 'somepackage'

ruby_block 'set myvar' do
  block do
    node.run_state['my_var'] = Mixlib::ShellOut.new('/bin/somecommand').run_command.stdout.strip
  end
end

As far as I know requiring 'mixlib/shellout' is not necessary.


How do you calculate the compound Service Level Agreement (SLA) for cloud services?

After reading Tensibai's excellent answer, I realised I used to be able to calculate this for network analysis purposes.  I dug out my copy of High Availability Network Fundamentals by Chris Oggerino and had a crack at working this out from, not quite first principals.
Taking my serial example directly out of Tensibai's answer is simply a case of multiplying the probability of each component being available by the other:

So

99.95% * 99.95% = 99.9%

Calculating it in parallel is a little more complicated as we do need to consider what the percentage unavailability will be:

The calculation is done as follows:

Multiply the unavailability of the two regions together.

0.1% * 0.1% = 0.0001%

Convert that back to availability

100% - 0.0001% = 99.9999%

Multiply the Traffic Manager availability by the availability of the two regions.

99.99% * 99.9999% = 99.9899%

The result is the whole system availability.

99.9899% is close to 99.99%

I ended up using Excel to perform the calculations, here is the values:

... and the formulas ...


Is FaaS required to have an official "Cloud"?

Google’s free computing course says:

  The US National Institute of Standards and Technology created a
  definition of cloud, although, there is nothing US specific about it.
  Here it is, cloud computing is a way of using I.T. that has these five
  equally important traits. First, you get computing resources on-demand
  and self-service. All you have to do is use a simple interface and you
  get the processing power, storage, and network you need, with no need
  for human intervention. Second, you access these resources over the
  net from anywhere you want. Third, the provider of those resources has
  a big pool of them and allocates them to customers out of that pool.
  That allows the provider to get economies of scale by buying in bulk
  and pass the savings on to the customers. Customers don't have to know
  or care about the exact physical location of those resources. Fourth,
  the resources are elastic. If you need more resources you can get
  more, rapidly. If you need less, you can scale back. And last, the
  customers pay only for what they use or reserve as they go. If they
  stop using resources, they stop paying.

This seems to be taken from the document at https://www.nist.gov/news-events/news/2011/10/final-version-nist-cloud-computing-definition-published

Using DevOps software with virtual machines, with or without containers, can happen without using cloud. In a similar manner, you can open a corporate account on a public cloud and not do DevOps at all. This happens more than we might expect when big companies are migrating legacy systems into the cloud where they simply have sysadmins or other Ops folks use the web interfaces to the cloud IaaS to manually commission VMs that they then manually configure. I have met teams that are convinced that they are doing DevOps when they are simply doing Ops with a few specialist Cloud Engineers who handle setting Virtual Private Cloud (VPS) meaning private connectivity into the cloud (e.g, VPN or a dedicated connection) and IAM (Identity and Access Management).

IMHO cloud and devops are most definitely not the same thing but are at their best when combined.


How to inject credentials to python virtual env in jenkins?

In order to pass masked credentials to your Jenkins job, do the following:

In Jenkins, install Credentials Binding plugin.
In your Jenkins, go to Credentials and click the global domain.
In the left pane, click "Add credential".
From the "Kind" dropdown, choose "Username with password".
Enter the username and password in the reflective boxes and choose an ID (you will use this ID when you want to access the credentials) and click OK.
Then, in your job, under "Build environment", check "Use secret text(s) or file(s)" and it will add a "Binding" section and an "Add" button, click the "Add" button and "username and password separated", choose a variable name to hold the username and another to hold the password and choose the credentials you created earlier from the credentials dropdown.
Then, in the "Build" section of the job, you can access the credentials using your chosen username and password variables.

I wrote an article about this subject in the past so feel free to check it for more detailed explanation and screen shots.


What are trusted providers that offer wildcard certificates and associated costs to be used in an Orchestration Platform like Kubernetes?

Have a look at "Ingress" and "Cert-Manager":

With a Kubernetes Ingress you can configure the load balancer, see e.g. https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer
With "Cert-Manager" you can get certificates from Let's Encrypt (like certbot does), and it handles renewing automatically. See https://github.com/jetstack/cert-manager


Should I give an assessment to DevOps Candidate

Yes, there absolutely should be an assessment, however it would be a good idea to agree upfront with one of the few DevOps people you know to validate the content and results.

Regarding the contents of the assessment, there are some questions still remaining after your edits. Key one is how your operations are done. Possible options are: you already have someone doing your server infrastructure and operations and looking for a purely CI/CD person, or you do not have good operations support (i.e. developers doing that and you're unhappy) and you need DevOps person to do server management too. Note, that if it's the later case, you'd probably need more than person over time but you need somebody senior to start with.

Note, that for larger organizations you'd find more specialization within DevOps (i.e. things like Release Management, DevSecOps and others), but for smaller companies those two - CI/CD vs Infra Operations are two main ones in my experience. You still need to keep those differences in mind if you're considering taking someone from a larger company.

Now, after answering those questions, you need to first establish the seniority level of a person you're looking to hire. If it's your first DevOps person - you would probably want somebody more senior, but it depends on other people in your existing team. Next you scope your assessment to the on-job tasks. My take - always give take-at-home assignments, I usually give it to candidates over weekend and I make sure they take no more than 3-4 hours.

For CI/CD case you may ask them for a simple CI pipeline in their tool of choice, built from scratch over some sample project. You should add extra questions about failure handling, parallelizing, caching, handling of automation tests.

For CI/CD + operations case you should add tasks about shell scripting, cloud security, terraform.

Also for the first DevOps / senior person it's always a good idea to add some questions about interacting with other teams. I.e. what developers would need to do and how the interaction process with them should be built.

Again, start with one of the few DevOps people you know and get them onboard as a consultant to assess and hire that first person, otherwise you're taking a lot of chances there - and DevOps is a pretty sensitive part of your business.


How to track (non ECS) container costs on EC2?

Since you are not asking about ECS containers, this means that you must have a bunch of EC2 servers that you manage in some way (ECS/Kubernetes/Nomad/etc...). These servers are most probably included in an Auto Scaling Group.

Once you run containers on these servers, AWS does not charge you for the container. AWS only charges you per second of EC2 instance time while the instance was in a running mode.

So in fact you could say that containers are "free", you don't pay for containers, you pay for the infrastructure underneath (EC2 servers).

Netflix ICE is a great tool to track EC2 instance costs, as well as the AWS Billing dashboard.


A container runtime is the part of your container environment in charge of the creation and basic features of your containers. The obvious one is Docker, but you can also find Containerd, CRI-O and other as runtime for your containers.

An orchestrator, by contrast, will not exactly create your container (ie, an orchestrator is not the technology used to create them). An orchestrator will use your container runtime to manage them. In a way, it's a layer around your container runtime wich allow finer usages, such as scaling, multi host deployments, load balancing...

The most famous orchestrators would be Kurbenetes, Docker Swarm...

I hope this help! Have fun.


How can you pass private environment variables to FARGATE tasks specified from a Docker Compose config

This is a well-written and up-to-date article which summarises the problem I've hit and confirms there isn't really a perfect solution at the moment in Fargate.

https://hackernoon.com/secrets-management-within-aws-ecs-1b6975819ccd - Connor Beardsmore

The alternatives listed are:

Bake into Docker - not an option with an open source repo
ECS Environment Variables - as I mentioned less secure
Docker Fetches from SSM Parameter Store - secure but "architecturally dirty"
Wait for Fargate SSM to be able to inject sensitive data!

Re. option 2. the problem is the environment variables are visible in the AWS console. But I wonder: if an attacker has access to your AWS console, you're pretty screwed already? So isn't this in fact an practically acceptable solution?


Trigger a job and get the job log on the same page in Jenkins

It looks like there exists a plugin to do exactly what you want: Display console output plugin.

However, I haven't tried this plugin myself, and it looks like this plugin is not well-maintained, so I cannot vouch for the quality of this plugin.


Docker container can't reach host services

It happens that the problem comes from ufw not allowing connections from docker containers on host ports.
Deactivating it helped for now, since it's only for dev purpose.


PowerShell command to automate Azure Linux VM

Apologies for not directly answering your question, but by the sounds of your question, you are trying to build an update management system to mitigate the threats of vulnerable operating system components and tools having unpatched vulnerabilities which expose you to risk.

This functionality is actually already available for both Windows and Linux via Azure Automation in the form of the unimaginatively named Update Management solution.

Essentially you will need to do the following:

Configure Azure Monitor and Log Analytics to provide information about vulnerable packages on your machines,
Ensure that any network security groups and firewalls are configured to allow access to/from the various components of the system,
Enable Change Tracking and Inventory,
Enable Update Management,
Onboard your machines.

I would also highly recommend employing Azure Security Center as this provides a clearer overview of vulnerabilities across your estate, not just as applied to specific virtual machines.