What is the best way to set up a CI/CD pipeline on ECS?

The question is a bit broad, but here are some suggestions to accomplish what I believe you are asking. If they are simple or not is for you to judge!
If the infrastructure referred to already exists and you would like to manage, update using infrastructure as code (IaC), you could try tools to convert the infrastructure to code. (These are new and at least one is in beta at the time or writing this)

Existing Infrastructure to Terraform:
https://github.com/GoogleCloudPlatform/terraformer
Existing
Infrastructure to CloudFormation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html

If the infrastructure does not exist, I recommend using infrastructure as code (IaC). AWS was mentioned, as well as the complexity of CloudFormation, with these considerations, Terraform could be a good option.

Cloud Posse offers some good modules to simplify using Terraform to build infrastructure. This module may be a good place to start.

Finally, if you would like to abstract away the bulk of the infrastructure and just run containers in the cloud.

Google Cloud Platform (GCP) offers Cloud Run, providing a completely managed service to run containers in the cloud with out much of the underlying complexity. The GCP Provider for Terraform can help here.
AWS Fargate cloud be another option, with Cloud Posse again offering modules for Terraform.

Once the code and infrastructure option is selected, CI/CD is as easy as adding a buildspec.yaml to the repository and adding the Code Build infrastructure to the Terraform code (AWS) and similar for Cloud Build (GCP).
Hopefully these options help find you the simple solution you are looking for!


Immutability is a term often used in computer science circles, which generally boils down to "not possible to change after creation".  It is typically used in reference to parallelism, concurrency, and thread safety.

The discussion of that topic is fascinating, but can generally be found elsewhere on Stack Overflow.  I'm resisting the urge to dive into it here.  The key concept is 'not possible to change after creation'.

Imagine if, on Amazon, you deploy a web service by baking it into a Machine Image (AMI - a pre-built instance you can repeatedly re-provision).  It connects to a backend database via credentials it gets out of a registry on startup.  It dumps logs into a logging tool like Splunk.  For normal day-to-day operation, you have no reason to ssh into this box.  If you need to scale up that service, you just create more instances of that AMI and adjust a load balancer.  Spinning it down is simply the destruction of instances and load balancers.

For the day-to-day operation, this box has no reason to change.  We can just fire up more off of the AMI.

What happens when you need to provide an OS-level security patch?  This is when you have a decision to make... do you bake a new AMI with the patch installed and redeploy all running instances, or do you ssh into existing images and update the patch?  There are plenty of people who would just ssh in.  The adherents of 'immutable architecture' just yelled at me for even suggesting such a thing is possible.

Immutabilists (if there is such a word) advocate baking the new ami.  They advocate removing all reason to ssh into a machine ever.  They advocate that any specific machine configuration should happen on the startup of that machine by pulling config details out of a repository.  This is the ultimate expression of 'cattle, not pets'.

Immutable architecture is specifically about machine configurations that have no reason to change after the creation of the machine image.  If something needs to change, bake a new instance image, shut down the old, bring up the new.


How do configuration management tools deal with someone messing up configuration files manually?

If the file is marked as being managed by a configuration management tool, it deals with someone messing up a configuration file manually being messed up by a hapless admin by clobbering the configuration file using something like Jinja or ERB templates

While this can often feel frustrating to an old-school UNIX admin when their change is clobbered in minutes or hours by a configuration management tool, (and I have seen some use things like extended file attributes to hack around this) it is far superior in that it 1) enforces a cattle-not-pets mentality and 2) self-documents a deployment - a factor which is vastly undervalued considering that 5-10 years later when an upgrade of a system is needed, hundreds of config files may need to be scoured such as a host file, limits files, logrotate files, syslog files, systemctl defaults files, firewalld/iptables config files, selinux files, cron files, and much, much more may need to be considered using the "pets" deployment philosophy.

9 times out of 10 this comes from a lazyness/unwillingness to learn a newfangled configuration management technology. That being said, I highly recommend that if you are going to manage a config file using a configuration management tool, you use Salt, Chef, Puppet, Ansible, etc to flag the fact that you have taken over management of said config file so that the admin is not completely blindsided whenever possible and as often as possible.

You can also support additive configuration - EG, I have used salt to manage rsyslog.conf and some files in /etc/rsyslog.d/ but not all files. This allows for a baseline set of defaults, while still supporting some customization without (barf) using a configuration management tool - thus allowing you to "have your cake and eat it too".


Is this an appropriate architecture, or can improvements be made?

My largest concern is around the MS SQL Server implementation. Coupling the read only instances so tightly to the services feels wrong. Is there a better way to do this?

Essentially what you have designed is a caching system - the service containers have a local copy of the data presumably so that for reads they don't have to make an extra network trip.

As you've pointed out, a more standard approach is to have a cluster of read replicas that all of the containers can read from.  This allows you to scale them separately from the application servers, which is good, because they generally need different things (do you really want to allocate large amounts of RAM to every application container?).  This will add network calls for database reads, but until that's proven to be an issue I wouldn't complicate the architecture to solve it.

If it does become an issue, a much more lightweight way of handling the problem is to run an actual cache locally, like memcache or redis.  You can tune the TTLs on individual objects to be appropriate, and it will automatically drop off rarely-requested data to keep the application server light.


How to help DevOps Engineers feel less like a lone wolf?

My first thought is "why is the he the only person doing ops, on a dev team,  especially when he gets to work with loads of automation?".  I think there's an opportunity there to address the lone wolf syndrome; particularly in a dev environment, infrastructure-as-code provides a great framework for sharing the load.  Ops people should be experts in determining and defining infrastructure needs for application, but they should also be able to build a platform to let dev roles do as much as they can independently.

It sounds like a silo within a team; old habits die hard.  A coder may not feel comfortable spinning up and hardening a server, but in a pure devops model, they should have the tools to do so.  An ops person in a devops team shouldn't responsible for delivering infrastructure for the app itself, but they should provide some insight into what is needed and some guidance on how the app developers can do it themselves.  It's almost a meta-infrastructure model; ops engineers build infrastructure that can build infrastructure on demand when requested by the development team.

Consultation, communication, and blending of responsibilities are all crucial to the success of a devops team.


kitchen verify hangs on centOS, but not Debian

I also had this problem show up.  I think it's related to SSH.  I was writing a custom sshd_config.  Vagrant login's were slow after the first converge and verify was taking forever.  Removing the sshd_config stuff from my cookbook fixed the verify and slow logins.

Maybe it's not very helpful, but it's a place to start looking, anyway.  Short term, I hacked it together:

unless ENV['TEST_KITCHEN']
  cookbook_file '/etc/ssh/sshd_config' do
    source 'sshd_config'
    mode '0644'
    notifies :restart, 'service[sshd]', :delayed
  end
end

Jye


How to explain devops to non-techy managers?

Being a consultant I am contractually obliged to answer, "it depends".  With that out of the way, I can actually answer your question.

What does it depend on? Well, that could come down to what your boss thinks about DevOps:

If your boss has heard of the term, via their obsession with CIO.com maybe, then ask them what they think it means.  From there work out what the difference is and if their view is compatible.  Identify a suitable project to trial DevOps on and pitch it to them.  Remember that at the core DevOps is culture so consider how that could be applied to a project.
If your boss has never heard of the term then build a business case for DevOps. Use the Puppet Labs State of DevOps and material from books such as The Phoenix Project to write the business case.  Find a problem that your boss has and DevOps could solve and use it as a conversation starter. As kenchew said you don't have to mention DevOps, you might for example, suggest that Operations be more involved in a project or more test automation to be planned as part of project delivery.
If your boss thinks DevOps is just another buzz word then do one of the above but absolutely don't mention DevOps, look at other similar models such as Site Reliability Engineering, Platform Engineering or Continuous Deployment and find out how they could solve the problem.

The key is to focus on understanding what your boss is motivated by, then carve out a small bit of time, money and people to take tangible steps to solving that problem.

I highly recommend the book To Sell Is Human by Daniel H. Pink, fundamentally Daniel Pink talks about how selling something is a very human thing to do, all we need to do is tap into the needs and align our "pitch" by proposing a solution that serves those needs.


Apart from your "this", there is also Compuware Source Code Download for Endevor, PDS, and ISPW Plugin. Some details about it (from the linked page):

  ... allows Jenkins users to download Endevor, PDS, or ISPW members from the mainframe to the PC. Source can then be accessed on the PC, for example, for SonarQube analysis and reporting.

So you may want to give "that" a try ...

PS: I'm assuming your question is about Endevor SCM, and not Endevor DB (that a completely different story, and a challenging one ...).


I would suggest adding a job that checks changes in the latest commit and build only that service where some files have been changed.


error while loading shared libraries: libtinfo.so.6 : two behaviors for same docker image

There is Docker's known issue #37531

To fix dependencies, you might try:

Either upgrade Docker to 17.02

curl -sSL https://get.docker.com/ | sudo bash

or install lib inside of container.
Since your container based on Centos (FROM centos), so try yum install

rpm -ql libXtst-devel-1.2.2-2.1.el6.x86_64


Heroku Slack integration without using Github

ChatOps uses GitHub status checks, so it's decidedly GitHub specific and won't work with another provider.

However, if you're just after notifications, what about Heroku's App webhooks? They have an 'entity' for new app releases (api:release) which might partially help you... in that you can set up a notification for successful releases, at least.

Alternatively, you could listen to the api:build entity's create event to know when a build starts, and the update event to know when its status changes - which I haven't used but presume would include a field for whether it succeeded or failed.

To subscribe to a webhook, you use the Heroku CLI:

heroku webhooks:add --include api:build --level sync --url https://example.com/hooks

This is outlined in more detail in the documentation here. The payload you receive will look like this.

This isn't going to get it straight into Slack for you though, so you'll need a piece of middleware to do that. Here's a tiny Sinatra app I forked from a fork. It converts Docker Hub notifications into the format Slack expects for an incoming webhook. It should be pretty easy to edit it to make it convert Heroku webhooks instead. Then, voila, the tiny app itself deploys on Heroku, and that's the URL you stick in the webhook subscription above.

This is probably a lot more steps than you wanted. Someone may have already done it somewhere... but I can say that at least doing it the above way will give you a lot more flexibility to create your own customised workflows where you need them!


This is a plugin that you can install.
Plugin Name: Build Pipeline
Plugin Web page: https://plugins.jenkins.io/build-pipeline-plugin/


Jenkins Pipeline project can't when on branch

This is the way to go:

stage('Deploy'){
  when {
    expression {env.GIT_BRANCH == 'origin/master'}
  }
  steps {
    ....
   }
}

Take care, this is only working with the declarative syntax.

The environment step is used to "set up the environment" meaning this is the place to declare environmental variables.

Also, in my case I did not declare the GIT_BRANCH var myself. But jenkins provides it as an environmental (therefore "env.GIT_BRANCH") variable if you did "checkout scm" to checkout your repository. If you want to check all the available env vars, do this as a step: sh 'printenv'

check out this article for more info: https://jenkins.io/blog/2017/01/19/converting-conditional-to-pipeline/


How to ensure that the docker build fails if an ENV has not been set?

An empty of non existent environment variable is the same thing, there's no reason Docker do not allow you to pass an empty variable as it could be a valid use case.

It's up to your responsibility to ensure needed environment variables are there and give an error message back when not.

Something along the line of:

RUN 'if [ -z "$AN_ENV_VAR" ]; then echo 'Environment variable AN_ENV_VAR must be specified. Exiting.'; exit 1; fi'

You can play with the exit code to automate some actions on failure around your docker build command.


How to have jenkins add a review to a pull request

I guess the simplest and direct solution would be to use GitHub API: https://docs.github.com/en/rest/reference/pulls#create-a-review-for-a-pull-request
But for the requirement you have, PR checks is usually the way to go and you have a plugin that does the API communication for you: https://www.jenkins.io/doc/pipeline/steps/pipeline-githubnotify-step/


Use one Helm chart for ALL microservices?

Generally, a Helm chart would cover all your microservices - ideally, you should be able to deploy whole application in one chart. It may get a little messy if you include external 3rd party dependencies, where you would need to handle dependent charts, but I assume this is out of scope of your question.
More so, if your microservices are very similar you could write for loops, such as:
{{- range $index, $service := .Values.myservices}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-{{ $service.name }}
# --- etc ---
{{- end }}

Now, where to put your chart - in most cases a preferred way is to have a separate repo for all Ops CD files, including Helm chart - so this is where Helm chart would live. This is not to be mixed with CI files, such as Dockerfile - those should live alongside microservice repositories themselves.


Suspend autoscaling processes through Cloudformation template

Got it from official documentation of UpdatePolicy . Indeed, we can suspend processes by using the UpdatePolicy attribute for the AWS::AutoScaling::AutoScalingGroup resource :

So instead of SuspendedProcesses, the right term is SuspendProcesses.
Also instead of putting SuspendProcesses under Properties, it should be under UpdatePolicy > AutoScalingRollingUpdate .

_

  MyAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      DesiredCapacity: 1
      MinSize: 1
      MaxSize: 1
      LaunchConfigurationName: !Ref MyLaunchConfiguration
      AvailabilityZones: !GetAZs
    UpdatePolicy:
      AutoScalingRollingUpdate:
        SuspendProcesses:
        - AZRebalance
        - Terminate


You can make changes in Main.tf as specified below:

resource "aws_s3_bucket" "b" {
  count      = "${length(var.domain_name)}"
  bucket     = "${element(var.domain_name, count.index)}"
  acl        = "public-read"
  policy     = "${file("bucket-policy.json")}"

  website {
    index_document = "index.html"
    error_document = "error.html"

    routing_rules = <<EOF
EOF
  }
}


Trigger downstream pipeline job for the branch via from upstream freestyle job

The "Add Parameters" can help you. You should set up your downstream pipeline with "GIT_BRANCH" parameter to build with that branch, and transfer it from this upstream job.


Startup of ECS Fargate task fails with ResourceInitializationError mount.nfs4 Connection timed out

Underlying reason:

The ECS service's network access security group did not have permission to access EFS.

Solution:

Add an inbound rule for type NFS in the security group as described in this tutorial: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html#efs-security-group


using bash script (or pipeline step sh), I would do:

JIRA_ISSUE_KEY=$(python my_python_script.py)
echo "$JIRA_ISSUE_KEY"


When to use Apache Airflow (MWAA) in AWS instead of AWS Batch?

AWS MWAA uses Apache Airflow to create workflows and DAGS (Directed Acyclic Graphs) with Python to orchestrate complex, dependent tasks. AWS Batch is, as the name states, a batch processing services that utilizes docker containers. Batch allows you to manage instance types and container sizes to fine tune costs of workloads. Batch unlike MWAA does not utilize any orchestration service, so apart from the queue that it utilizes, there is no way to control the workflow of tasks.
If you have a job that requires complex workflows, AWS MWAA will reduce the complexity of managing those workflows. If you require large scale containerized processing for image analysis, data conversion, etc., AWS Batch will allow you to easily control your workloads to optimize speed and cost.
It should also be noted that AWS Step Functions is also a very efficient way to orchestrate various types of processing and application tasks. For simple tasks, there is very little overhead and management compared to AWS Batch or MWAA.


How do I get Openshift to let Tomcat retain it's hosted site's changes

You need to use a Persitent Volume.

Your /path_to_data/openshiftdata is probably on a disk that belongs to the pod and is being destroyed each time.


Considering you say you want to build an image (and not push it to Docker hub or wherever):

docker-maven-plugin project recommends using dockerfile-maven for new projects, as the former

  [...] ultimately led to a lot of unnecessary confusion with our users that stemmed from introducing extra abstractions and a need for configuration on top of what Docker is providing.

The dockerfile-maven README should get you started, with both a sample Dockerfile and  the relevant plugin section for your pom.xml


Are there any truly simple artifact repositories?

I generaly work with a CDN, you can upload files and compressed files, and many CDNs has api ,  like it, but exist many other not conventional solutions, by example insert code or script in a fake image using base 64 and upload it to server public , or use facebook pixel to add script or file. Hope this help you.


ansible: difference between a variable and a fact

Ansible facts are data collected about the (target) systems on which Ansible takes actions. They are variables, but set by Ansible (in a way like system defined variables). They are collected during Gathering Facts stage of a playbook run, and it is controlled by the gather_facts setting. Ansible calls this variables discovered from systems. That said, it is possible to set custom facts also.
Some examples are:

ansible_hostname - the FQDN of the target system
ansible_os_family - the Operating System family of target system (RedHat, Debian, etc.)

The other variables are the ones we can set as per our requirement (in a way like user defined variables).
Some examples are:

my_fav_fruits: [ 'orange', 'apple', 'banana' ] - yours might differ.
config_dir: '/etc/my_app/conf.d' - for my application configuration files.

Update:
Updating answer to make it relevant to the edit(s) made in question.
As @Bruce Becker's answer rightly pointed out, there is a difference in the precedence of variables set with set_fact. Also variables that need to be set at "run time" can be set this way. Without further explanation, taking your example variables, if I create a play like below:
- hosts: localhost
  vars:
    nginx_ssl: '/etc/nginx/ssl'
    nginx_conf_file: '/etc/nginx/nginx.conf'

  tasks:
  - name: set nginx path to /opt when running on Debian
    set_fact:
      nginx_ssl: '/opt/nginx/ssl'
      nginx_conf_file: '/opt/nginx/nginx.conf'
    when: ansible_distribution == 'Debian'
  - debug:
      msg: 'ssl: {{ nginx_ssl }} and conf: {{ nginx_conf_file }}'

Then the set_fact variables will take precedence (on Debian) and output will be:
"msg": "ssl: /opt/nginx/ssl and conf: /opt/nginx/nginx.conf"

On other distributions, it will be what was declared in vars:.


Test coverage of changed code in pull requests in git-tfs

IMHO from the workflow prospective code coverage measurements are fundamentally similar to builds, smoketests or any other types of QA checks and can thus be performed in similar manner. I can see several options:

include them in the PR verification path, just like builds and/or smoketests, similarly positioned relative to the code review stage.
include them as a step in the pre-commit gating system, if you use one - most effective IMHO, especially for large projects where scalability comes into play (mentioned in my answer to How do you scale your integration testing?).
include them in the CI pipeline, maybe on a parallel execution path if they're lengthy, to not add too much to the total CI execution time. Very useful to use these runs as up2date baselines for results comparisons for the other 2 cases listed above.

I'm not familiar with MS tools, but if MSTest doesn't offer good integration with the other tools in your workflow you could build custom wrappers to implement a typical submit/execute API with a pass/fail result (or set of results) which most tools can integrate with.


saltstack: archive.extracted template jinja

There is no much choices regarding your situation, I see two of them :

You ask your dev dpt to add your jinja templating to their builds (which may be hard or even impossible)
You can use salt module such as file.append or file.replace to add / replace your jinja code inside (this is not very clean that's the least we can say)

If you'd share an example that may help us to give some good advice


how can I make jenkins find 2 jenkinsfile in a repo with multibranch?

You need to update your multibranch pipeline build configuration to use Jenkinsfile.ci and Jenkinsfile.cd instead of Jenkinsfile. And you would have to create two multibranch pipelines, one for Jenkinsfile.ci and other for running Jenkinsfile.cd.


What package managers are supported on Microsoft hosted agents with CMake enabled?

According to this page, the hosted agent supports NuGet, npm, and Maven.

It looks like it might also support pip.
You could install cget using pip and then use cget to install your package. A bit of a roundabout way, but it might do what you need.

"C:\Program Files\Python36\Scripts\pip" install cget
cget install nlohmann/json

I don't have a way to test this right now, but I will try some real examples and edit tomorrow if I have time.

Since it is a CMake/Git project, you could also just clone the repository (or add it as a submodule) and build it every time. Since I often use libraries that aren't managed, I tend to take this route and leverage the caching that many CI platforms offer to keep my build times low. Sadly, it looks like VSTS hasn't gotten to this yet.


How to decrypt Jenkins passwords from credentials.xml?

Luckily there is a hudson.util.Secret.decrypt() function which can be used for this, so:

In Jenkins, go to: /script page.
Run the following command:

println(hudson.util.Secret.decrypt("{XXX=}"))

or:

println(hudson.util.Secret.fromString("{XXX=}").getPlainText())

where {XXX=} is your encrypted password. This will print the plain password.

To do opposite, run:

println(hudson.util.Secret.fromString("some_text").getEncryptedValue())

Source: gist at tuxfight3r/jenkins-decrypt.groovy.

Alternatively check the following scripts: tweksteen/jenkins-decrypt,
 menski/jenkins-decrypt.py.

For more details, check: Credentials storage in Jenkins.


How can I load multiple paths into the workspace during docker run?

You can use Docker volume mounts at run time.  For example:

$ mkdir -p app/src logs
$ touch app/src/foo.c logs/bar.txt
$ docker run --rm -v $PWD/app:/app -v $PWD/logs:/app/logs alpine ls -l /app /app/src /app/logs
/app:
total 8
drwxr-xr-x    2 1000     1000          4096 May  6 23:33 logs
drwxr-xr-x    2 1000     1000          4096 May  6 23:33 src

/app/logs:
total 0
-rw-r--r--    1 1000     1000             0 May  6 23:33 bar.txt

/app/src:
total 0
-rw-r--r--    1 1000     1000             0 May  6 23:33 foo.c

This mounts two directories on the host into the docker container.  Notice in this example that inside the container, they are mount on inside the other.


Is local / staging / production still industry standard?

When you say "Industry Standard" that's very much going to depend on what industry.
You can drop code into production however you want (legally restricted industries obviously being the exemption to this), but the question you should probably be examining is if you should be doing that.
I've recently worked on products which range from having incredibly test coverage, huge amounts of infrastructure set up to perform every imaginable regression test against any integration point that exists, and even though it was not the done thing with that particular product, after passing all of the automated testing required for a branch to be mergable, I probably would have been pretty comfortable lodging straight to production. On the other end of the spectrum I've also worked on projects which arguably were developed in production given that the production instances were so dissimilar to what was actually checked in to source control or running in any test environment and, of course, no test cases to be seen.
The standard approach to making production deployments as safe as possible is to try and ensure that the thing you are planning to deploy is as rigorously tested as possible, and to do that most of the testing will need to be automated, otherwise you'll just end up spending your life testing.
Additionally, most modern systems use standard patterns to make a production roll back an extemely frictionless task, if not to automate it entirely. If you are deploying canaries of your software, or you are running a blue/green deployment and can have built in mechanisms to roll back if certain success conditions aren't met, then it further reduces the risk of a production change breaking something.
Essentially, I would question anyone telling you how to manage and derisk your production releases unless they are just as intimate with the software as you are.


How to change a statefulset in OpenShift?

Shameless self-answer:

So it turns out the only way to edit the statefulset, and this is by design, is to delete it and recreate it with the new values.

Doing that does not delete or stop the pods or the the PVCs - those will re-attach to the new statefulset spun up later, so you won't lose anything. N.B.: you can delete the pods, and the PVCs will still stick around. When the pods belonging to this statefulset restart later, they will re-attach to the PVCs.

There is a discussion on one of the OpenShift/Kubernetes task trackers (which I cannot find anymore) that explains the rationale: in contrast with stateless Deployments, where OpenShift can simply scale up a new set of pods, then scale down the old one, while routing traffic to the new one, this is not easily possible with statefulsets, since those have persistent PVCs, so there usually is a very defined number of pods that can/should run at a time.

So, if OpenShift would do an upgrade for you, due to an edit, it would need to decide how to do that (spawn up new pods parallel to the old one would require spinning up PVCs which may wreak havoc with your quota; replacing them in-place could wreak havoc if your pods need to talk to each other to synchronize). Hence, they require you to do this on your own. For example, you can decide to delete and recreate the statefulset, then delete one of the pods after the other; they will be automatically recreated immediately, as usual. You can then wait until the next pod has reached a state where it is synced with the other ones (in whatever way your software does it), then delete the next one etc., until you have upgraded them all.


Managing state (e.g. presence/absence) in Ansible, any established best practices?

This can be done with two tasks with loops:
# tasks/packages.yml
- name: pkgs | Ensure Good
  package:
    name: "{{ item }}"
    state: present
  loop: "{{ packages['present'] }}"

- name: pkgs | Ensure Bad
  package:
    name: "{{ item }}"
    state: absent
  loop: "{{ packages['absent'] }}"

This will loop over all of the items in the list that you define and ensure that they are either present or absent.
You could also use a shorthand:
# tasks/packages.yml
- name: pkgs | Ensure Good
  package:
    name: "{{ packages['present'] }}"
    state: present

- name: pkgs | Ensure Bad
  package:
    name: "{{ packages['absent'] }}"
    state: absent

This would run faster, since you don't have a list of tasks, but a task with a list, but you would lose some visibility into which packages you're actually ensuring the state of.
Caveats:
Ensure that there is no overlap between the two lists. One way would be to make an assertion on the lists before you run the task:
- name: ensure no overlap between good and bad
  assert:
    that: "{{ packages['present'] | intersect(packages['absent'] }} == [] }}"

Also ensure that the lists are not empty.
To answer your final question:

Q: are there best practices about managing the state of "entities" on the configured infrastructure?

It is usually my practice (I wouldn't call it "best", but take it for what it's worth) to declare the explicit state of a resource. It's quite difficult to do this exhaustively for packages that should be absent - this is something akin to maintaining a "blacklist". One approach might to be start from a policy and implement specific controls in it. IE for ssh config ensure that protocol v1 is absent, etc.


All volumes are deleted when EC2 intance is deleted via terraform

I could see some spelling mistakes in the below mentioned line, rest all are fine i think.

delete_on_termination = "${var.delete_on_termincation}"

Better you could avoid this line, This might be deleting the EBS when u terminating the ec2 instance. Skip this option and check.


What should I use for configuration management on AWS?

A good way to find comparison information about things like this is googling for "X vs Y", e.g. "Chef vs AWS stacks", "Chef vs Puppet" or something like that. That does turn out subjective information, and while it is nothing like having hands-on experience, you still get a few nice nuggets here or there.

For example, Chef gives you the full Ruby language while Puppet has a ruby-based DSL. Does that matter for you? Only you can tell.

From my experience with Puppet/Ansible (and I don't know Chef yet) it's all mostly down to taste. For example, I really like the fact that Ansible needs only a working ssh key and nothing else. And I like that there is no central repository. Others may like the fact that Puppet has a repository. Also, I like the dead simple configuration file format. But that is all subjective.

So, to cut to the chase, my advice would be:

Use AWS Stacks if you are fully committed to AWS, will never change to another provider, and like the higher integration with AWS features. It uses Chef internally anyways, and exposes the Chef files to you.
Use Chef directly if you like to learn something new and want to stay a bit vendor independent. If you later decide to commit to AWS fully, then you should be able to do so.
Use Puppet (which AWS starts to support as well, seemingly) if you, well, like Puppet more, or if you can't stand Chef.


Failure to build with docker-compose: Invalid signal: SIGQUIT

Latest edge release 2.3.1.0 fixed this and many others problems


How to raise awareness about the security risk involved with running Docker?

A tool that I think is rather under recognised is s2i. The basic idea is that you don’t put a Dockerfile into your source repo specifying the runtime to use you keep your app code bare (just code). The decision as to which security patched s2i image to compile and run the code with is a separate matter that is independent and can be centrally managed.

The way this is used in the OKD world is that a “BuildConfig” pulls your code from git to make an application image and is trigger by either a git webook or a docker registry push that updated the s2i image with a security patch.

In our case we have many node and php apps. To security patch either runtime we push the latest s2i image into our OKD docker registry and that causes all our application images to be recreated and redeployed.

Why I think that is of interest is that it basically says that what the code is, and how the code is run from a container security perspective, should not be conflated. They are two concerns that are not on the same cadence. A high impact vulnerability should be patched in the main test environment immediately and not be dependent on talking to several scrum teams about pushing a change to the Dockerfile in each team’s git repo.

When you have many apps or many small dev teams that separation to do “policy based security patching” of containers can be very useful. You can run s2i on your desktop or with container based SaaS ci tool such as circleci. It would be straightforward to run it in, say, Jenkins. That means you can have, say, dozens of micro service Jenkins deploy jobs triggered by git webook that run s2i and push the final image. You can then control the version of the s2i image you use in all the Jenkins jobs in one place. You can then script a security update that updates the s2i docker url to the latest security fix and triggers all Jenkins release jobs to run to apply the patched s2i to all microservices. No code or config needs to change in any git repo.

While I mention OKD by default it doesn’t let you run a container image as uid 0 (root). It uses a random uid and gid 0. This is for security in case the process gets out of its jail and tries to modify the host. Most public docker images just expect you run as root and file permission don’t let you run the app in the image as an arbitrary uid as docker —user ${rand_uid}:0. You have to bake an image that gives gid 0 the file permissions to run the app. The s2i images all work under these circumstances.

If you need apps in docker to lookup the use details in /etc/passwd when running as a random uid the you can fake that. Here is a script that runs git in docker as random uid where git wants to lookup the current in /etc/passwd so the script fakes it.

The lesson here IMHO is that while in theory docker gives perfect isolation in practice all code is subject to security flaws. In theory many small dev teams will update all Dockerfiles immediately or very frequently but in practice dev teams are focused on features not long term security patching. (I work with one microservies platform that currently has >200 git repos and dev teams on three continents so coordinating anything is a challenge.) So security should be “defence in depth” and take concrete steps to do centralised patching of container images, not run containers as root, and use a random uid that isn’t in the host /etc/passwd so cannot read or write outside of its jail.


Should performance tests be run using a CI and how often?

I will assume by CI you mean one integration server. The best way to get the maximum out of your tests and environment stability is to decouple environment per test type. The best example is production which is lively "tested" by your users.

Be aware that by environment, I mean the server or servers used to deploy a specific version of your application or applications (I will refer this as the build). In such emvironment, you would tipically have similar setup as your live/production environment. Based on your team restrictions, here is what I proposed:

Restricted from having a new environment

Simply schedule a job to start load testing when the traffic is minimal. This is usually at night. Upon test completion, a report should be available and granular enough to indicate where improvements are required based on your internal NFR threshold.

No restriction on having another environment

Create a new environment for load tests
For each new build, if it was successfully deployed and/or tested in the CI server, deploy to the new environment
Run the load tests and save the results. Depending on your CI tool, you might be able to configure reports and alerts according to your results quality

In both case, you will need to specify your load test suite in a way that allows you to figure out where latency is too high. I would personally suggest you the second solution. Though it requires more setup and maintenance, it is the most benificial for the following reasons:

Tells you exactly when the build starts to be unstable. For the first solution, you might need to investigate  because a lot of change is happening during the day.
High visibility on what/who made the system slower. At this time, the teams are aware of the issue and can remedy right away. The nighly job might be overlooked on the long run.
Can serve as a performance gate where all builds with a certain response time will stop from going further (i.e. production) until the response time is improved. This is unpractical to put in place with a nightly job. Also, you would need to wait a day to see the result of an improvement or run the test manually.

If you are in the journey to CI/CD, fast failures is prefered as delayed failures. Losing a day to fix a problem can have considerable impact on your time to market.


How do I troubleshoot missing data in my Prometheus database?

I think your can do some kind of alerting on a metric rate with something like this:

ALERT DropInMetricsFromExporter
  IF rate(<metric_name>[1m]) == 0
  FOR 3m
  ANNOTATIONS {
    summary = "Rate of metrics is 0 {{ $labels.<your_label> }}",
    description = "Rate of metric dropped, actually: {{ $value }}%",
}

The main idea is to alert whenever the metric rate is at 0 for 3 minutes, with the proper metric name and a label somewhere telling from which exporter it comes it should give you the correct information.

Choosing the right metric to monitor by exporter could be complex, without more insight is hard to give a better advice out of vacuum.
This blog post could be an inspiration also for a more generic detection.


Blacklist network ranges Docker is allowed to use for its containers (swarm mode)

Docker should avoid using a network if there's a route to that network (see ip r). You can whitelist networks that docker is allowed to use with configurations in several places:

You can adjust bip in /etc/docker/daemon.json which controls the docker bridge network named bridge. This needs to be set to a gateway ip with CIDR notation.
You can adjust default-address-pools in /etc/docker/daemon.json which is a list of CIDR blocks for all other bridge networks.
You can pass --default-addr-pool when initializing a swarm to a CIDR block for overlay networks.

Make sure the two pools are large enough to create multiple networks, e.g. a /16 subnet to allow creation of 256 /24 networks.

More details of this are available in my DockerCon presentation (press P for presenter notes).


How do I load a Jenkins Shared Library in a Jenkins Job DSL seed?

It's possible to use jobDSL from Pipeline. (Im using multibranch pipeline as it allows to configure pipelineTriggers)
You can configure your seed job to be a pipeline like this:
def gitCredentialsId      = 'github-jenkins'
def jobsRepoName          = 'https://github.com/my-jobs-repo.git'
def sharedLibraryRepoName = 'https://github.com/shared-library-repo.git'

properties([
   pipelineTriggers([githubPush()])
])

pipeline {
    agent any
    stages{
        stage('Seed Job') {
            agent any
            steps {

                checkout([
                    $class: 'GitSCM',
                    branches: [[name: '*/main']],
                    doGenerateSubmoduleConfigurations: false,
                    extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'shared-library']],
                    submoduleCfg: [],
                    userRemoteConfigs: [[credentialsId:  gitCredentialsId, url: sharedLibraryRepoName ]]
                    ])

                git url: jobsRepoName, changelog: false, credentialsId: gitCredentialsId, poll: false, branch: 'main'
                jobDsl targets: 'jobs/**/*_job.groovy', additionalClasspath: 'shared-library/src'
            }
        }
    }
}

This way we can configure our seed job to checkout shared libraries repo into workspace subdirectory shared-library and specify additionalClasspath for jobDSL groovy scripts
So in your groovy scripts you can simply use import without @Library annotation


Different download speeds from AWS S3 CLI vs Console/Websites

TL;DR
Try reducing the number of concurrent connections used by awscli to 1 using this command: aws configure set default.s3.max_concurrent_requests 1

You could be experiencing an issue with the number of concurrent connections that the awscli is opening, if you are using the aws s3 commands (not aws s3api).

In the cli user guide under the Managing Objects section you will see that all high-level upload commands on an S3 Bucket beyond a large object threshold (awscli default 8MB) will use a multipart upload.

I couldn't find anything about the use of concurrent connections when using the web console, but looking at the output of lsof I could see my browser was only opening a single connection for a ~500MB file download, whereas using aws cp 10 concurrent connections were being opened. So this would appear to be the key difference in behaviour - at least with my browser and test case.

Running a quick, dirty test with the default (10) connections and then 1 concurrent connection, my download completed in the following times:

s3.max_concurrent_requests=10   8m22.965s
s3.max_concurrent_requests=1    6m47.772s

So it seems the overhead of the extra connections can sometimes cause slower downloads/uploads over my network, rather than making them faster, which could be what you are experiencing.

You could also try tweaking some of the other defaults around how multipart operations are handled which may yield better performance over your network, listed here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html.


What does "Does not have minimum availability" in k8s mean?

As @Tensibai indicated in one the comments, this could be caused as there is insufficient CPU or memory, but that is not always the case.
For example, a helm chart was just deployed, it failed and the workload in GCP indicated that:

Pod errors: CrashLoopBackOff

Based on the comment of @Tensibai the first impression was that there were insufficient resources, but further analysis using kubectl describe pod <pod-name> indicated that in this case the livenessProbe check failed:
Liveness probe failed: Get http://10.16.0.13:80/: dial
tcp 10.16.0.13:80: getsockopt: connection refused

In summary, the Does not have minimum availability message is generic. Multiple issues could trigger this and more in dept analysis is required to find the actual error.


Jmeter simulating traffic and avoiding 429

Given that this is symptomatic of rate limiting in Nginx in this case, the solution is to configure the rate limiting feature of Nginx to allow this

A whitelist could  be created in the configuration to remove ratelimiting from the url generating the traffic.

To follow the Nginx documentation, this could be specified as whitelisting an IP.

The current server setup is :

limit_req_zone $binary_remote_addr zone=flood:10m rate=12r/s;
limit_req_zone $binary_remote_addr zone=bot:10m rate=200r/m;
limit_req_status 429;
limit_conn_zone $binary_remote_addr zone=connperip:10m;
limit_conn_status 429;

This sets up the rate limit on connections. A whitelist can be created as suggested above, by simulating a boolean condition with a key value map:

geo $limit {
    default 1;
    10.0.0.0/8 0;
    192.168.0.0/24 0;
}

map $limit $limit_key {
    0 "";
    1 $binary_remote_addr;
}

This removes the rate limit (sets it to 0) if the remote address falls into the whitelist:

limit_req_zone $limit_key zone=req_zone:10m rate=5r/s;

server {
    location / {
        limit_req zone=req_zone burst=10 nodelay;


GitHub Actions Runner execute commands with sudo GitHub Actions

airtower-luna from GitHub community answered it here and it worked for me
Answer pasted below:
The command in your workflow file is not the same as the one in your sudoers file, sudo doesn’t know they are functionally equivalent. Use twentydb.service in the workflow, too.
Also, is the user your runner runs as in the sudo group? If not, that entry won’t apply to it. However, I’d suggest using a separate entry for the user rather than adding the runner to the sudo group.


[Converting my comment to an answer]

One way to do it would be to write the logs to some external file, and then having a task after it which makes use of failed_when condition, and remove the log file, if the previous task was successful.

Something like this should help you.

 - name: Run Py script
      command: <>.py  > <>.log
      become: yes
      register: PyScript
      ignore_errors: True

    - name: PyScript on success
      command: rm <>.log
      when: PyScript|succeeded

Note: This might not be the best way to handle your problem. But, this was a hack which helped me do my logging and monitoring.


Back-filling prometheus (and related system) metrics?

Answering my own question (after a few days of research).
I have yet to test this but https://github.com/Cleafy/promqueen seems to be designed for this purpose:

PromQueen made possible to record prometheus metrics offline. PromQueen can, > therefore, backfill the recorded data inside a native prometheus database.
PromQueen is composed of two primary tools:

promrec tapes the metrics in a specified output file.
promplay backfills the prometheus database from scratch.

It doesn't have a lot of contributors but it comes from another POC repo and it was updated in the last 6 months, so I suspect it'll work fine.
In any case, it proves that you can plausibly back-fill prometheus, so it's not impossible.
Roadmap - Prometheus
You can also find this in the prometheus road-map, so it's clearly slated to be built-in eventually.
https://prometheus.io/docs/introduction/roadmap/

Backfill time series
Backfilling will permit bulk loads of data in the past. This will allow for retroactive rule evaluations, and transferring old data from other monitoring systems.


What are the limitations of Puppet in comparison to Ansible?

There are of course several pro and cons for each of Puppet, Ansible, Chef and add your favorite tool here as well. So I'll try to stay away from opinion, and share what is great in Ansible as a matter of fact.

The main capability that puts Ansible above the others is not having to rely on some custom/additional agent running on the target nodes, instead being founded on ssh connections only. Yes, it still requires an ssh server, Python and a bunch of Python libraries on the nodes, and if your distro of choice (or, good luck, there are some windows nodes) does not ship with them, it will be a little bit painful to bootstrap. But that's unlikely, and may well even make you think again about your distro.

That will simplify monitoring, not eat additional resources, not force the system to run a daemon as root all the time, and in general feels better inside the UNIX philosophy. Chef has chef-solo, Puppet can be run master-less, but they both work "the other direction," by cloning and via hooks respectively. While with Ansible, a merge in the source repository can trigger the deployment in a fashion we all are comfortable with, be it in Jenkins, in the git master, or in some other tool like Rundeck for instance.


How to buy more Azure DevOps pipelines build minutes?

You can set up billing in your Organization settings in DevOps

https://dev.azure.com/~your organization name~/_settings/billing


What are the different job permissions possible in Jenkins, and what are the reasons you would use each?

There are 4 type of user permissions in Jenkins:

Open

Jenkins is accessible for everyone with no permissions. Basically anyone can do anything. This should be used for internal use only.
User authentication (all admins)

All users are admins. The authentication method can use Jenkins’ own user database, LDAP, Unix user/group database, servelet container (e.g. Tomcat or Glassfish user DB) or other authentication plugins. This can be used when there is no need to restrict users from doing certain actions.
Matrix based security

This is a global permission per role/group security strategy where each user where permissions are associated with groups to which users belong to. This is the most common authorization to use for multiple users.
Project based matrix

It is authorization strategy where user permissions can be configured per individual jobs. Once activated, permission matrix can be activated on each job page. Useful when specific jobs shouldn't be accessed by certain users.

Related page: Securing Jenkins - Access Control.


To replace all lines in the file replace module is needed. This is the equivalent (backup differs).

  tasks:
    - lineinfile:
        path: myconffile
        backup: yes
        state: absent
        regexp: '\(ADDRESS = \(PROTOCOL = TCP\)\(HOST = 10.0.0.1\)\(PORT = 1501\)\)'
    - replace:
        path: myconffile
        backup: yes
        regexp: '\(ADDRESS = \(PROTOCOL = TCP\)\(HOST = 10.0.0.2\)\(PORT = 1501\)\)'
        replace: '(ADDRESS = (PROTOCOL = TCP)(HOST = my-site.com)(PORT = 1501))'
    - lineinfile:
        path: myconffile
        backup: yes
        state: absent
        regexp: '\(BALANCE = yes\)'

The simplified version is below.

  tasks:
    - lineinfile:
        path: myconffile
        backup: yes
        state: absent
        regexp: "{{ item }}"
      loop:
        - 'AAA'
        - 'DDD'
    - replace:
        path: myconffile
        backup: yes
        regexp: 'BBB'
        replace: 'CCC'


What is self hosted version control system?

Let's take it word by word:

Self: You or your organisation owns this thing. By contrast, a cloud-based offering by GitHub.com, Azure, GitLab.com, etc. (see this list on Wikipedia for a more comprehensive list) is not owned by you, but rather the service provider in question. Do not confuse ownership of the code (dictated by it's license – typically owned by you), with ownership of the service for performing version control on the code.
Hosted: You or your organisation are responsible for operating and maintaining this thing. In some cases, this is exactly the same software (e.g. one can self-host GitLab or run GitHub Enterprise “on premise” in your own datacentre). This means keeping it secure, making any changes to it, serving user requests for change, etc. You are also responsible for some part of the infrastructure necessary to provide this service. This may vary, depending on who you have service provision agreements with, from everything down to the power and networking, or only the last layer of the service (here, the version control system itself), or anything in between.
"version control system" (or VCS): anything (software or otherwise, but most likely software) which provides the functionality necessary to control versions. Note there is a subtle difference between Version Control and Version Control Software, although the meaning is often conflated.

The term VCS can refer to any version control system, even when it is not applied to software, however in our context, it can be assumed that you are referring to a version control system applied to software.

In other words, a “self-hosted VCS” can be thought of as:

  A service and all of its components which is under your own ownership and control, acting as a version control system for software.

Often, this can take the form of:

base infrastructure (network/power/server),
operating system,
authentication and authorisation mechanism, and
the VCS server itself (Git, SVN, etc.).

The second part of your question is about how it is different from GitHub. The following contrast can be made:

Functionally there may be differences between what you can do with GitHub an what you can do with your self-hosted VCS. GitHub, as is the case for almost all VCS hosted by third parties (i.e., not hosted by you) goes a few steps further than simply doing version control. There are collaboration tools, identity management, issue tracking, etc. built into their service. While this is strictly speaking not part of VCS, it greatly improves the experience of using one, almost to the point where doing version control without them feels impossible.
GitHub owns and operates the service. You may make suggestions to change their service, API, etc., and perhaps even contributions to their open source products, but you do not own the service. It is provided to you based on the SLA for the offering that you choose. You cannot make your own changes to it. This is overall a very good thing, unless you are in the business of providing the VCS itself – but chances are that you just want to use the service, not compete with it.

Update to answer the edited question:

  does it get uploaded to the server, or it is saved on my local machine? If it is uploaded to the server, then if I reset the computer, will I able to access my code again?

You are conflating two issues here: self-hosting and distributed version control. Of course, if you self-host on your own computer, your run the risk of losing everything if the machine dies. However, of someone else hosts your code in only one place, then you still risk losing everything if their machine goes down.

To address this (and several other issues related to centralisation of source control), distributed version control was invented. A good discussion is in the git book.


How can I eliminate the need for an Ansible control machine?

The solution is to automate setting up the control machine, i.e., the VM. You can use, for example, Vagrant, Terraform or similar tools (I'll stick with Vagrant in this answer for this example, it's about the principle). With that approach, the VM is wholly defined by a text file ("Vagrantfile"), and can easily and repeatably be created and re-created. The Vagrantfile species a base image, for example some particular Linux distribution; networking etc.; and necessary command lines to set it up, going from there.

You can set up your VM (i.e., install Ansible) by just providing all necessary shell commands. In this particular example, this is probably enough. If you need a more complex VM, you can get kind of recursive, and configure the VM itself by using Ansible (or one of its alternatives).

Obviously, you will want to treat the Vagrantfile the same as your other configurations; i.e., commit it to your SCM and keep it up to date.

This is just an example: You don't need to use Vagrant or a VM; you can also install your Ansible inside a Docker image or whatever your cloud provider of choice provides as containerization.


How to get GitLab CI to rebuild on JIRA issue status change?

I didn't get what's wrong with webhooks
You just create a webhook on Gitlab CI side and provide this URL to your Jira account.
https://gitlab.com/api/v4/projects/PROJECT_ID/ref/REF_NAME/trigger/pipeline?token=TOKEN

PROJECT_ID - unique project id on Gitlab
REF_NAME - your qa / stage / beta branches within repository
TOKEN - uniquely generated access token (gitlab provided)
Look here for more details how to trigger build on Gitlab via this call


Why are books so prevalent in the DevOps community?

In most cases, the recommended books are not about technology. While technology changes, the fundamental principles behind organizations like system thinking, leadership, common sense, etc... do not change as often.

Books such as The Goal, and even The DevOps Handbook do not mention much technology on their pages but rather ways to manage work being performed by people.

Many problems are technology related, topics such as Microservices, Architecting large-scale systems, Infrastructure as Code, etc... these don't talk about a specific tool and/or technology but rather about an architectural topic. A field of knowledge that people who build large systems need to know in order to build the system correctly. This knowledge is rare, and its great the books are written about these subjects - just disregard the tools mentioned, or translate into their new-er reincarnation.

One of the better books about creating quality software (imho) is Agile Software Development, Principles, Patterns, and Practices. And while the language used in this book (Java) has moved along quite a bit, the examples provided in the book are timeless and can be easily translated into any other language of choice.

Some of the problems the DevOps movement tries to solve are involved with common ways work is managed in organizations that just doesn't make any sense. As Eliyahu Goldratt often said (author of The Goal) "Common sense is not very common".

These books teach principles of thinking correctly about problems and human relationships in a system setting so that the whole system is improved. The lessons are old, and unfortunately, only rarely there are people who work in the field who actually learned them.

Naturally, there are also authors who wrote books about such and such fizz-bang tech tool that is new and relevant to the field, like AWS or Docker or Jenkins or whatever and just want to push their book sales ... but I try and exclude these kinds of blog posts from my answer.


Jenkins executes Selenium test in headless mode while use `Jenkins as Service`

Following command line if you plan to run the same application in both a headless and a traditional environment: more details

java -Djava.awt.headless=true
You can add environment variable in Jenkins configurations
Manage Jenkins -> Configure System -> Global Properties -> Environment Variables section As:
Name :  DISPLAY
Value:   0
Installing Jenkins master as a Windows service. This will run browser in other users account to get rid from this do following things:

3.1 Go to your services and stop Jenkins service. To see all your services run
services.msc, from there find Jenkins service and disable\stop it.

3.2 Go to jenkins home directory(C:\Program Files(x86)\Jenkins) and start
cmd with administrator mode and hit
java -jar jenkins.war

if still your problem persist go through following threads
1. for as service,

2.  for general queries


Why does the jenkins-cli.jar linting functionality not work?

After upgrading Jenkins the issue was solved. It turned out that the issue was version related.


Porting a docker container to a different Linux host OS

Yes, as long as you have the Docker daemon running on the CentOS. Docker heavily utilizes a set of kernel features - namespaces and cgroups. That set comes native for all Linux derivatives.

In terms of 'porting' the image, you have 2 widely used options:

docker save on your Ubuntu host. Get the saved archive to your CentOS and load the image with docker load
The above option is not so convenient for the standard CI/CD stack, so you might just push your image from your Ubuntu host to a so called registry. Once the image is in your registry, you could pull it from your CentOS host.


Gradual rollout management for client applications

It sounds like you are talking about Canary Analysis ? If so, Spinnaker has deployment based on  canary analysis built on, using metrics provided by the application telemetry. To quote the Spinnaker docs:

Your application might send performance metrics which are published and available by default. You can also install a monitoring agent to collect more comprehensive metrics, and you can instrument your code to generate further metrics for that agent. In any case, you need to have access to a set of metrics, using some telemetry provider, which Kayenta can then use to make the canary judgment.
Support is built in for Stackdriver, Datadog, Prometheus, Signalfx, and New Relic.

So, you use Spinnaker as a continuous deployment service and instrument the application with one of the supported mechanisms. Canary analysis can then be done to manage the roll-out of new version.


Is it possible to containerize using docker without a local setup?

Checkout Play with docker here. If you just started then it is best to install docker on your local machine to get more of a native experience and understanding of docker networks and overlays etc... and after that you can move on to any cloud like aws ECS service, azure etc....


How to get rid of develop branch for simplified Git flow

Let's say you take out the master branch  (you can rename develop to master to confuse your team if you like later) and simply use tags for releases either on develop or hotfix branches. You took out a branch, but the difference is just a change in syntax. Change for change sake.

Now let's say you actually take out develop with keeping the locked master branch. What will happen is that integration of code will slow, it will live longer separated in feature branches, especially close to release dates. This will Increase the difficulty of merging each time and slow the process down.

Within the constraints you have laid down I don't see any positive effects of making such change. It would require relaxing the constraints, especially the first one.


Docker: Let containers communicate to each other but don't expose to internet

If you're looking to access your database only from localhost, you can do something like that:
    ports:
      - "127.0.0.1:5432:5432"


Is it possible to mount an s3 bucket as a point in a docker container?

No you can't.

S3 is an object storage, accessed over HTTP or REST for example. Just as you can't mount an HTTP address as a directory you can't mount a S3 bucket as a directory.

Having said that there are some workarounds that expose S3 as a filesystem - e.g. 's3fs' project. How reliable and stable they are I don't know.


What is the simplest way to list which attached disks are encrypted (and which not) on AWS?

To get the number of non encrypted volumes you can run this command:

aws ec2 describe-volumes --region <your_region> --filter "Name=encrypted,Values=false" --query "length(Volumes[])"

length will return the length of the array Volumes flattened by the selection operator [] (more details on JMESPath documentation).
As we filter the slection for non encrypted volumes (--filter "Name=encrypted,Values=false") this should allow to demonstrate to the auditor the number is 0 not encrypted volumes.

Same filter can be applied in the console, in the ec2 page, under 'Elastic Block Store' => 'Volumes', type Encrypted : Not Encrypted to filter the view to non encrypted volumes only. you may add Attachment Status : Attached to list only attached volumes.


I'm still fairly new to the Docker game, too, but I've got a few little projects that I do exactly what you are asking about. Through my own experience with trial & error (mostly error...) I would suggest using a docker-compose.yml with your network set to static internal IPs so each container will have predictable, repeatable results and IPs that you can configure your containers to communicate across.
I'm not familiar with BaseX but I have a sample situation using docker-compose version: "3.7" here that could easily run multiple apps and/or be added to or repurposed to fit your hypothetical.
This example docker-compose has the following:

A separate container for populating my app-volume volume with the contents of a public GitHub repo.

An Apache Web Server container with PHP-FPM enabled for a "selective backend" that gets built from a Dockerfile in the ~/dockerbuild directory.

An nginx web server container used as a reverse proxy for all things PHP to be passed to Apache (but serves & caches pretty much everything else).

A MariaDB container in the mix for the app DB that I populate with a mysqldump file after the app gets built.

This is a slimmed down version of my docker-compose.yml, so note that you can add a variety of things like healthcheck: and command: to fit your needs.
You execute with docker-compose up -d --build to both build and start your app in one command (but you'll probably want to create your own ba(.sh) file that you execute afterwards, too for those important finishing touches).
Take notice of static-network. And while I am using named volumes in the example, that's just a preference of mine.
version: "3.7"

services:

  github_repo_clone:
    image: debian:latest
    container_name: github_repo_clone
    networks:
      static-network:
        ipv4_address: 172.20.0.254

    command: bash -c "
        apt-get update &&
        apt-get install git -y &&
        rm -rf examplerepo &&
        git clone https://github.com/exampleuser/examplerepo.git &&
        rm -rf /var/www/vhosts/exampleuser/examplerepo &&
        mkdir -p /var/www/vhosts/exampleuser/examplerepo &&
        cp -rapv examplerepo/* /var/www/vhosts/exampleuser/examplerepo &&
        rm -rf examplerepo &&
        tail -f /dev/null
      "
    volumes:
     - app-volume:/var/www/vhosts

  apache_php-fpm_backend:
    container_name: debian-build-demo0
    build: ~/dockerbuild
    ports:
     - "8080:8080"
    networks:
      static-network:
        ipv4_address: 172.20.0.2
    volumes:
     - app-volume:/var/www/vhosts

  nginx_rp:
    image: nginx:latest
    container_name: nginx-build-demo0
    ports:
     - "80:80"
     - "443:443"
    networks:
      static-network:
        ipv4_address: 172.20.0.3
    volumes:
     - app-volume:/var/www/vhosts

  mariadb:
    restart: always
    image: mariadb:latest
    container_name: mariadb-build-demo0
    ports:
     - "3306:3306"
    networks:
      static-network:
        ipv4_address: 172.20.0.10
    environment:
     - MYSQL_ROOT_PASSWORD=4007p@$$w04d
     - MYSQL_USER=dbusernamegoeshere
     - MYSQL_PASSWORD=@p@$$w04d
     - MYSQL_DATABASE=dbnamegoeshere
    volumes:
     - app-volume:/var/www/vhosts
    healthcheck:
      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"]
      timeout: 10s
      retries: 5

volumes:
  app-volume:
#    external: true

networks:
  static-network:
    driver: bridge
    ipam:
      driver: default
      config:
       - subnet: 172.20.0.0/16

My clean up command when I take the app down is kind of ugly, but not terribly difficult to read and goes something like this:
docker-compose down -v ; docker rmi -f dockerbuild_apache_php-fpm_backend:latest mariadb:latest nginx:latest debian:latest ; docker rmi -f $(docker images -q --filter "dangling=true") ; docker rmi -f $(docker images -q --filter label=stage=intermediate) ; docker volume prune -f;
I think each "independent app" could (and probably should) have its own docker-compose and/or Dockerfiles, but you may find synergies with certain Dockerfiles that you can create custom command: instructions for (such as the git_repo_clone example) to be used & shared across multiple "apps".


DC Jenkins server to AWS servers connection

Depending on your use-case, you should look either:

Establishing a persistent VPN connection to the AWS VPC using the Site-To-Site VPN as described here: https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html
Using an OpenVPN client via the build jobs using the client VPN service as  described here https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/what-is.html


How to deploy a Kubernetes service using NodePort on AWS?

More often than not you should not be using NodePort.

You're in AWS so you can consume the AWS Cloud Provider. Try using ingress resources or LoadBalancer service types. NLBs can do TCP/UDP port passthrough if that's what you need.

Edit: After further review, it looks like your just exposing a web server (I noted port 8080). NodePort is 100% not what you want. Look at Ingress


Either request_timestamp needs to be converted from the ISO 8601 formatted string into a timestamp with time zone, or current_timestamp needs to be converted into the string.

One method is to convert request_timestamp into a string in format %Y-%m-%d, e.g.

split_part(request_timestamp, 'T', 1)
substr(request_timestamp, 1, 10)

then compare it with current_timestamp which can be converted to the ISO 8601 formatted string.

Here is the final query:

SELECT *
FROM elb_logs
WHERE split_part(request_timestamp, 'T', 1) = split_part(to_iso8601(current_timestamp), 'T', 1)
LIMIT 10


In a multistage Docker build, should the WORKDIR be deleted?

When creating an image using a Dockerfile, docker layers are created for each step. The description of layers can be found on the official docker page.
Accordingly, when using Multistage to create an image, it introduces a new build step that can be named using an alias in the FROM statement, and all layers except the image layer created via the last FROM statement are deleted.
After build your image, check if the layer that contains WORKDIR whether exists or not using docker image inspect. I think you probably know the unique layer value for each step.
You can see the image consists with which layers.
e.g.,
$docker image inspect [OPTIONS] 464503b73bfe # IMAGE
[
    {
        "Id": "sha256:464bf1...46c",
        ...

        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:26711...8fcf",
                "sha256:6c5a51...56e0"
            ]
        },
        ...


A comparison of idempotence and immutability

The two terms are very different.

Let us start with immutability, which literally means "no mutations" or "no changes". In the DevOps sense, it means that once you created an artifact, be that a container image, or a VM image, or maybe a package from compiled code - you declare that you will never ever change it. Often if any changes are required, you declare that a new version of "thing" will be created instead.

The term idempotence means that when changes are applied multiple times, the state is mutated (changed) just once. First, it already assumes that there are going to be changes applied which means that you cannot have something both immutable and have idempotent actions done to it (no actions are done to it by contract).

In the use of configuration management tools, idempotence is used in some cases when applying the same change multiple times. Like adding the line that says localhost to the /etc/hosts file, you don't really need multiple such lines and if one already exists it is safe to not try to append again.

Also idempotent is a term used to describe actions that attempt to change things, while immutable is used to describe nouns (objects) that are set against changes done to them.

Why is an immutable object useful? Because when you copy it around, for example from a dev environment to qa to production. You already know quite a lot (but not everything) regarding how it is going to behave. In many cases, the parts that are working will be consistent, and the parts that are broken are going to be consistent as well.

Why are idempotent actions useful? Because when you want to change a state of some object, in many cases it is useful to only verify that the change has been applied and apply changes just in case it is required. For example, when a configuration item in a file is missing or has the wrong value, it is useful to add it just once or change it just once while applying the action multiple times. In many other cases, like log files, you don't want to have idempotent actions because you often do want to append another line each time some event happens.


What are the different types of work in a development team?

Nice question!

Possibly, Changes are further strategic product development beyond current work which is more than required by technical debt? For example, introducing a new framework, or nice abstraction layer. (which could break a lot though)

Internal projects could be custom helper tools to automate something invisible in the backlog. Sometimes you have also to fill in time sheets.

By the way Unplanned work could be also a request to support sales or HR or some unexpected meeting or all hands social skills training event.

There is anyway a sequel to "Phoenix Project" called "Unicorn Project" by same author, focusing on the Dev part of the equation.

I haven't read it myself yet but hope you might find the answer there.


Directory structure - how do I upload it as-is to an artifactory?

This can be done through the Jrog CLI. The CLI has a recursive option that looks like it will do what you hope.
jfrog rt upload --recursive artifactory-mirror/* artifact-repo/dir-struct/

The CLI also has some performance improvements for uploads that doing it through curl or a browser don't seem to get. Also, scriptable!
To include full directory structure you will want to use the flag --flat=false
jfrog rt upload --flat=false --recursive artifactory-mirror/* artifact-repo


Is there a puppet module to manage policykit rule on Ubuntu 16.04?

It appears that there is. For future reference, the best place to look for puppet modules generally is to search Puppet Forge. That is the central repository for modules and any module uploaded there can be installed using puppet by typing:

puppet module install jhoblitt-policykit

But obviously, you will want to update the name of the module to be whatever you want to install.


How to create named volumes in Kubernetes?

k8s have documentation page about volumes and their types https://kubernetes.io/docs/concepts/storage/volumes/

So in short you need to define spec.volumes and spec.containers[].volumeMounts like:

apiVersion: v1
kind: Pod
metadata:
  name: test-pd
spec:
  containers:
  - image: gcr.io/google_containers/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /cache
      name: cache-volume
  volumes:
  - name: cache-volume
    emptyDir: {}


Try directly accessing https://my-api-lb-111222333.us-east-1.elb.amazonaws.com from your web browser and you'll get a similar error.  The SSL certificate is not just used for encryption -- it also verifies that the host is not an impostor, by being consulted in a check of whether the hostname used to access the target matches the hostname of the certificate that's offered by the target

Because amazonaws.com is not your domain, you can't get a certificate for that.

You need to assign a domain name from your domain to the ALB in DNS, e.g. origin.example.com, and point API Gateway to that name, not to the system-assigned hostname of the ALB.


Configuration of AWS CodePipeline for Android CI/CD

The indentation looks incorrect for the artifacts section in your buildspec-test.yml file. It is nested under the phases section. That way it is not being output properly and can't be picked up by the next build.

It should be the same as in your second file.

version: 0.2
phases:
  build:
    commands:
      - $ANDROID_HOME/tools/bin/sdkmanager "build-tools;27.0.3" "platforms;android-27"
      - sh gradlew assembleDebug
artifacts:
  files:
    - app/build/outputs/apk/debug/app-debug.apk
  #some other stuff here.


By default, there is no way to visualize skipped stages in the Jenkins UI.  See this issue report on the Jenkins JIRA about visualizing skipped stages.
There are a few workarounds that I'm aware of:
You can disable the Groovy sandbox to allow importing the aforementioned class.
You can also use the imperative-when plugin for scripted Pipelines as described in this blog post to add visualization (note that I haven't tested this personally).
Or you can skip the visualization altogether and wrap your build steps in an if clause to conditionally execute them.  You'll see in the Jenkins UI that a stage takes 0ms to execute if it has no steps, so there's at least some indication that it was skipped even if it doesn't explicitly say "skipped".


What is the difference between the traditional Development and Operations Model and Site Reliability Engineering?

Thankfully, since Site Reliability Engineering developed internally at Google and only recently has started to make its way into the broader community, it is fairly well-defined.  What isn't, though, is web operations (or "systems administration" - as an example of the lack of clarity, you use both in your question).  It's difficult to discuss the differences between two things when you're not altogether sure what one of them is.

But I'm an adventurous fellow, so I'll give it a shot.

In very traditional shops, developers and sysadmins are very siloed from each other.  The devs build an app, then consider their job complete as soon as their code has been committed.  The sysadmins take the build artifacts (which may be just the code, if it's an interpreted language) and deploy it to production servers.  It's the sysadmins' job to keep the application running smoothly, and in general manage the production environment.  However, often performance problems come from architecture issues in the app; the sysadmins don't have the programming knowledge to know what the app is doing, and the developers don't know how the app acts in the production topology with production traffic, so no one is equipped by themselves to solve the problem.

Additionally, the developers are usually judged on how quickly they can produce new features, while the sysadmins are judged on how infrequent the app breaks in production.  Since change is one of the leading causes of breakage, this puts the two departments at odds with each other - an old rivalry that hurts the business and the people involved.

At some point, some developer-centric companies got so annoyed at this that they began practicing "NoOps" - they eliminated their operations departments and the perceived roadblocks that came with them.  In reality, this meant that developers took on operations roles, but maintained their old titles.

In a discussion surrounding NoOps, John Allspaw, then VP of Technical Operations at Etsy and an editor of the well-respected Web Operations book, defined roles at Etsy this way:

  Etsy Operations is responsible for:

  Responding to outages, takes on-call
  Alerting systems thresholding, design
  Architecture design and review
  Building metrics collection
  Application configuration
  Infrastructure buildout/management

  Etsy Development is responsible for:

  Responding to outages, takes on-call
  Alerting systems thresholding, design
  Architecture design and review
  Building metrics collection
  Application configuration
  Shipping public-facing code

  Neither of those lists are comprehensive, I'm sure I'm missing
  something there. While Etsy Ops has made production-facing application
  changes, they're few but real (and sometimes quite deep). While Etsy
  Dev makes Chef changes, they're few but real. If there's so much
  overlap in responsibilities, why the difference, you might ask? Domain
  expertise and background. Not many Devs have deep knowledge of how TCP
  slow start works, but Ops does. Not many Ops have a comprehensive
  knowledge of sorting or relevancy algorithms, but Dev does. Ops has
  years of experience in forecasting resource usage quickly with
  acceptable accuracy, Dev doesn't. Dev might not be aware of the pros
  and cons of distributing workload options across all layers1-7, maybe
  only just at 7, Ops does. Entity-relationship modeling may come
  natural to a developer, it may not to ops. In the end, they both
  discover solutions to various forms of Byzantine failure scenarios and
  resilience patterns, at all tiers and layers.

In his world, developers and ops engineers had very similar high-level skill sets and responsibilities; where they differed was in their expertise.  Their differing specialties encouraged them to work together to solve problems, and their common base-level skills gave them a language in which to do that.

This is generally the definition of web operations that I land on for most cases.  So it's the one we're going to continue along with.

So then, what is Site Reliability Engineering?

The Google SRE book opens with a definition of SRE... and then another one... and then spends a chapter continuing to define the role and an entire book covering the specifics.  Even when developed in one organization, it seems that it's difficult to condense the job down to one single agreed-upon definition.

To start with, we need to walk back to 2003, when Ben Traynor joined Google and founded what came to be the first Site Reliability Engineering team.  Recall that a few paragraphs ago we were in the early 2010s; but in 2003, the industry was still pretty set on the sysadmin/developer divide as the natural way of things.  So when Ben says that SRE was what would happen if a software engineer created an operations team, this was a much more radical melding of the two worlds than it appears now.

The definition given in the preface emphasizes each of the three words individually:

Engineering - the use of computer science and engineering concepts to solve problems
Reliability - a focus on making systems more scalable, more reliable, and more efficient
Service - the later evolution of "site", emphasizing that SREs are responsible for networked services

The introduction chapter lists the tenets of Site Reliability Engineering as:

Ensuring a durable focus on engineering - taking pre-emptive action to avoid frequent pages and other "toil"
Persuing maximum change velocity without violating a service's SLO - a subject that can easily have its own several-hundred word answer, but roughly summarized as helping developers make changes, as long as they don't cause too many issues
Monitoring - automatic alerts when things go wrong
Emergency response - fixing things when they're broken
Change management
Capacity planning
Provisioning
Efficiency and performance - ensuring that a service performs at an expected level - bottlenecking hurts users, but excess capacity costs money

I'd categorize Site Reliability Engineering as a specialized subset of modern Web Operations.  An SRE organization focuses heavily on automating everything, to a degree that is only cost-effective in fairly large companies.  Ideas like error budgets can only work when your service has many, many requests, as otherwise you lose granularity (for a smaller service, a particular error could affect 0-20% of your requests, depending on the minute).  Related areas like security are absent from the SRE definition because companies large enough to have true SRE teams have dedicated teams for security.

The SRE program, as defined by Google, is web ops developed for the specific needs of Google, and not necessarily applicable elsewhere.

However, Site Reliability Engineering has been expanding in broader industry use recently.  My current job title is an SRE, even though I work at a much smaller company and my job description fits pretty well with John Allspaw's 2012 Etsy web ops definition.  My theory is that we've been progressing through titles as a shorthand for espousing the evolution of a single field:

We started as sysadmins.
Then as web sites became more of a "thing", job postings started to refer to web operations engineers to distinguish sysadmins who specialized in the web from those who also handled general office IT.
Then DevOps was supposed to separate out those who were comfortable using programming to reduce their web ops workload.
But as DevOps got muddled by the lack of a clear definition, we adopted Site Reliability Engineering to specify that we're looking for people who are on-call supporting production services.

So what is the difference between a sysadmin and an SRE?  The year in which they received their title.  What is the difference between traditional operations and site reliability engineering?  SRE is merely the current incarnation of ops, using new tools (hello, containers!) and, as networked programs continue to become more larger and more important, an increased focus on practices that allow one engineer to do more.


WIndows or Ubuntu server when developing on windows?

I dont see downsides of using your Windows laptop. Its a matter of preference. If you know how to deal with Windows Environment variables, paths, EOL characters, you should be good.
Many developers use Mac or Linux because they deal better with *unix like systems or because similar to the server they will run their apps.
At the end of the day after you build your MEAN solution the output should be the same regardless the OS.
Windows Server or Ubuntu server are just a OS to 'professionally' run your app. Absolutely no need to spend extra money to develop a MEAN app on those OS, I recommend to stick your your computer for development.


How to exclude a node in kubernets using podAntiAffinity?

To exclude a set of nodes when submitting a job in kubernetes.

you should use Node affinity which is conceptually similar to nodeSelector –and will allow you to constrain which nodes your pod is eligible to be scheduled on, based on labels on the node and you should be able to use hostname.

On below example i run a job to 5 completions and all are started on node-01 of my cluster with node affinity defined.

Example : job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  labels:
    run: busybox
  name: busybox
spec:
  completions: 5
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: busybox
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - k8s-node01
      containers:
      - image: busybox
        name: busybox
        resources: {}
      restartPolicy: OnFailure

Result :

ubuntu@k8s-master:~$ kubectl get all -o wide
NAME                READY   STATUS      RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
pod/busybox-2gdp6   0/1     Completed   0          3m20s   192.168.85.204   k8s-node01   <none>           <none>
pod/busybox-l654d   0/1     Completed   0          3m55s   192.168.85.201   k8s-node01   <none>           <none>
pod/busybox-n7dbr   0/1     Completed   0          3m29s   192.168.85.203   k8s-node01   <none>           <none>
pod/busybox-pfc79   0/1     Completed   0          4m11s   192.168.85.200   k8s-node01   <none>           <none>
pod/busybox-r4xln   0/1     Completed   0          3m16s   192.168.85.205   k8s-node01   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   31h   <none>

NAME                COMPLETIONS   DURATION   AGE     CONTAINERS   IMAGES    SELECTOR
job.batch/busybox   5/5           60s        4m11s   busybox      busybox   controller-uid=e12f136b-d828-4f7b-a49b-303499496a8a

I had two worker nodes on my cluster on job yaml i defined affinity to node01 only.

$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   31h   v1.17.2
k8s-node01   Ready    <none>   31h   v1.17.2
k8s-node02   Ready    <none>   31h   v1.17.2


Kubernetes installation, configuration and verification in a local environment

You probably don't want to delete the file, just remove it from your path.  You will need to look through your System and Account environment variables and look for PATH and remove C:\Program Files\Docker\Docker\Resources\bin\.

There is a good chance it that that will just work, I have both Docker Desktop and gcloud installed on this computer and I don't have the folder above in my PATH and both products work just fine without any warnings about old tools hanging about.

If you do continue to run into problems, for example with MiniKube, it can be helpful to use the Get-Command PowerShell command to figure out which executable is getting called:

Get-Command kubectl | Format-List

Will give you the full path to the executable that would be run if you issued the command kubectl:

Name            : kubectl.exe
CommandType     : Application
Definition      : C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe
Extension       : .exe
Path            : C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe
FileVersionInfo : File:             C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe
                  InternalName:
                  OriginalFilename:
                  FileVersion:
                  FileDescription:
                  Product:
                  ProductVersion:
                  Debug:            False
                  Patched:          False
                  PreRelease:       False
                  PrivateBuild:     False
                  SpecialBuild:     False
                  Language:


Although the comments have highlighted the fact that this playbook is fine, as long as it is designed to do what it needs to do:

  The reason behind this is that, the first play will create ansible user with sudo access. Here, I have used root user to run the play. Then I have to run another play as ansible user.

In the first play, you have only one role: role1. If in role1 you have a task that creates the ansible user, then you can use that user to connect the next time. This is actually something many people do, including myself, to put a known baseline on a service:

- name: Ensure Ansible user is present (RedHat)
  user:
   name: ansible
   comment: "ansible user created by bootstrap playbook"
   generate_ssh_key: yes
   groups: wheel
 tags:
 - bootstrap
 when: ansible_os_family=="RedHat"

- name: Ensure Ansible user is present (Debian)
  user:
   name: ansible
   comment: "ansible user created by bootstrap playbook"
   generate_ssh_key: yes
   groups: sudo
  tags:
  - bootstrap
  when: ansible_os_family=="Debian"

- name: update sudoers to ensure ansible user can sudo
  lineinfile:
    dest: /etc/sudoers
    state: present
    regexp: '^ansible'
    line: 'ansible ALL=(ALL) NOPASSWD: ALL'
  tags:
  - bootstrap

This will ensure that the user is added, and included in the right groups, with sudo rights -- it is just an example of how you could achieve this, there are other strategies of course.

  My question is how to run these play one after another? I first want to run first one then after that second one.

That's exactly how playbooks work.

  If I run the playbook then it will get stuck because it will first run first play and tries to run second play where it gets stuck.

If you do not have a task to configure the user required in the next play, then yes, it will fail (not get stuck).
The point is that since Ansible is idempotent, you don't have to worry about skipping the play because if the user is properly configured, the first play will simply have no effect.

However, to answer the actual question

  How do I skip a play in an Ansible playbook ?

This is done using the conditional execution using when


Jinja syntax error: missing endif, in spite of providing it

There is no error in what you have posted here. The error lies elsewhere in your code which you did not include in the OP. Using the Jinja live parser, when pasting in:
{% if spade.corename == 'FI' and spade.nodetype == 'master' %}
dataimport-script:
  file.managed:
    - name: /opt/bin/DataImport.py
    - source: salt://files/DataImport.py

dataimport-crontab:
  file.managed:
    - name: /etc/cron.d/solr-dataimport
    - contents: |
                # # set up data import for FI every 2 minutes
                */2 * * * * root /usr/bin/python /opt/bin/DataImport.py
{% elif spade.nodetype in ['slave','ds'] and spade.corename == 'FI' %}
update-fi-solrconfig:
  file.replace:
    - name: {{ salt['pillar.get']('solr:home_dir') }}/data/{{
salt['pillar.get']('spade:Corename') }}/conf/solrconfig.xml
    - pattern: '"autoDeletePeriodSeconds">30'
    - repl: '"autoDeletePeriodSeconds">-1'
{% endif %}

With a pillar of:
spade:
  corename: FI
  nodetype: master

Everything renders just fine. This tells me there is nothing wrong with your snippet. The fact that your error indicates you have at least 205 lines of code, but the example has far less tells me the issue was with another "if" block further up in your code.


In addition to the good answer by @therianthropie, here are useful notes to help you understand the situation better.:

You can set requests and limits very arbitrarily (and yes, as fractions of a CPU and very small amounts of memory).
Kubernetes will not do its job well if you don't get real values for these.
If you under-shoot your request, kubernetes will schedule a lot of apps without requesting enough resources and you'll have problems with the CPU/memory/disk, and so on.
If you over-shoot your request a lot, your kubernetes nodes may be doing virtually nothing.
Some apps are very  CPU bound, some memory bound, some use virtually no resources.  So, you really can't guess.
Prometheus is a popular tool that you can install into kubernetes with basically a single command.
You can chart things in grafana.
So, then you choose your arbitrary  values, then watch the charts as you use the app, and then you can adjust them to something sane / useful so kubernetes can do its job well.  You can also set alerts with something called alert manager if your values are too ridiculous (it's a good idea once you're ready).
Note - requests are what kubernetes allocates for your pod.  Limits are the max resources your pod can spike to when it needs them. There is a general assumption that not all pods will spike in usage at once / or over a long term or else this idea would not work.


How do I use the remote-exec provisioner with Terraform?

Terraform Provisioners are a last resort, so I'd encourage you to think about other options first.
In this case, your example suggests that you're already using cloud-init and so if at all possible I'd suggest modifying that cloud-init configuration to include a request to install the package you want to install. If you are using the cloud-config YAML format then you can use the Package Update Upgrade Install module to get a similar effect to your yum install command line:
packages:
 - coolprogram

However, if you've eliminated all other options except provisioners then the typical way to use remote-exec with an EC2 instance is to pass one of the instance's own IP addresses as the hostname, which you can do by using the special self object in the connection block to refer to the attributes of the object that the provisioner is running against, like this:
resource "aws_instance" "example" {
  ami           = "ami-1234"
  instance_type = "t2.large"
  key_name      = "foobar"
  user_data     = local.cloud_config_config

  connection {
    host = self.private_ip
  }

  provisioner "remote-exec" {
    inline = ["sudo yum -y install coolprogram"]
  }
}

The above example uses self.private_ip, which will work only if the host where you are running Terraform is running in or able to connect to the VPC where you're deploying this instance. Another option is to use self.public_ip to use the public IP address, but that will work only if your VPC has an Internet Gateway and your instance belongs to a security group that can accept incoming SSH connections from the internet.
In addition to supplying a suitable IP address, you'll also need to generate and assign an allowed SSH public key to the EC2 instance and provide the corresponding private key in the connection block, using the private_key argument. (Or one of the other authentication strategies that a connection block supports.)
These connection and authentication complexities are, as you hopefully saw in the Terraform documentation I linked earlier, one of the main reasons why provisioners are a last resort. Using cloud-init instead will avoid this because cloud-init runs automatically on system boot and retrieves the user_data string using a private API accessible from the instance's private network interface, and so it doesn't need any outside connectivity, to start any listening services, or to generate any new credentials.


Kubernetes on k3s can't resolve domains from custom dns server (fritz.box with dnsmasq)

I believe this is an current bug in k3s that upstream DNS is hardcoded to 1.1.1.1.  this should be resolved shortly https://github.com/rancher/k3s/issues/53


How to make a Jenkins API call when using "Log in with Google" as security?

Typically you authenticate API calls (I assume you mean HTTP REST API; there is also a Groovy API) using an API token.  The API token can be generated from a user's profile page while logged in as that user (Profile page > Configure > API Token).  More information about generating and using API tokens is available in the official documentation.  There is also a question about API tokens posted to StackOverflow.


What is the ec2 cli command to modify the minimum number of nodes in a scaling group?

I'd go with aws autoscaling set-desired-capacity following the documentation

But if you're up to make a button you should look into the various AWS sdk (on top right is the list of them) to call the related API instead of forking to a system command.


Kubernetes API Server KUBE_SERVICE_ADDRESSES parameter and Pod Network configuration

https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"