Retaining at least one successful build in Bamboo

IMHO if your successful build doesn't push to an artifact repository, there's a problem in your pipeline.

Jenkins or Bamboo do not aim at keeping built artifacts forever and they don't give you any versioning delivery, as such that's a point of failure in your delivery system.

I'd advise to review your method to use an artifact repository and that will solve this XY problem of keeping a successful build inside the build system used as an artefact source.

Note: I can't speak for Bamboo's team, but adding this capacity involves raising the code complexity for a 'side usage'. As such it can makes sense Bamboo's team avoid it to avoid adding code complexity for a feature not in their overall goal. I don't think they avoid it to enforce a best practice, but more to avoid raising unnecessary (from the product point of view) complexity to their code base.


What are best practices for creating and triggering various pipelines regards monorepos?

If this is translated to a monorepo that means that 1 change results
  in 3 pipeline triggers, while in some cases only 1 pipeline should be
  triggered.

I would definitely take this point into further consideration. One of the most compelling reasons for using a monorepo is that it does not create a technical separation of projects. There is only a logical separation, with the only technical distinction possibly being your versioning (unless you have a universal version number). This means, that you have no 100% guarantee that any one project does not depend on another project. Because of this, your final pipeline at some point should include all of the projects within your repository. By doing so, you are guaranteeing that a change to one project does not negatively impact any other projects in your repository (at the cost of the project with the longest time to build, publish, etc.).

That being said, if there are some steps in your pipeline that you would like to run separately (lint, syntax, compile, etc.), most CI's should have a way to filter your repo for a specific folder/project. If it doesn't, and you still don't want to run against all projects, you may need to consider refactoring your repository into separate repos or changing CI tools to comply with your workflow.


Ok tried to understand your problem exactly... This is what is happening:

$ echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq | grep value
jq - commandline JSON processor [version 1.5-1-a5b5cbe]
Usage: jq [options] <jq filter> [file...]
[rest of output is omitted for brevity]

The important thing to notice here is the usage line (emphasis is mine):

  Usage: jq [options] <jq filter> [file...]

The filter is not optional, if you don't give one, jq tries to parse the rest of the command line as its filter and throws an error. A workaround for your case here is:

$ echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq '.' | grep value
  "value": "New",

which indeed greps the "value" line as the filter passed is '.' - this just pretty prints the json content, but that's far from the best use of jq. If you wish to get only this line it would be better do:

$ echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq '.value'
"New"

If you wish to get the output without quotes then you can add add -r option to jq like so:

$ echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq -r '.value'
New

from jq --help:

  -r     output raw strings, not JSON texts;

that's a little introduction to jq, that's probably not enough to solve your problem at all but as you didn't specify your problem I can't help more than that.


Is making kubernetes masters schedulable a really bad idea?

https://kubernetes.io/docs/tasks/administer-cluster/highly-available-master/

  Best practices for replicating masters for HA clusters

  Try to place master replicas in different zones. During a zone failure, all masters placed inside the zone will fail. To survive zone
  failure, also place nodes in multiple zones (see multiple-zones for
  details).

  Do not use a cluster with two master replicas. Consensus on a two-replica cluster requires both replicas running when changing
  persistent state. As a result, both replicas are needed and a failure
  of any replica turns cluster into majority failure state. A
  two-replica cluster is thus inferior, in terms of HA, to a single
  replica cluster.

  When you add a master replica, cluster state (etcd) is copied to a new instance. If the cluster is large, it may take a long time to
  duplicate its state. This operation may be sped up by migrating etcd
  data directory, as described here (we are considering adding support
  for etcd data dir migration in future).

I would like to emphasize the following quote

  A
  two-replica cluster is thus inferior, in terms of HA, to a single
  replica cluster.

If less then 3 nodes will be used then it is basically more safe to run 1 master, but if one goes down there will be an issue as well. So it is better to run at least three masters.


Scheduling collisions possible in IaC - Infrastructure as Code - solutions vs. software defined data center?

I disagree with your definition, kubernetes, openshift and other are scheduler and as such should go in the SDDC part. IaC things are Vagrant/Terraform/Cloudformation/Vmware Cloud management. They are about writing code to define the infrastructure.

With this in mind k8s may fit both, a deployment.yml is IaC to drive the k8s SDDC.

Let take exemple for a k8s cluster running on top of vSphere infrastructure:
You have a scheduler k8s on top of another scheduler vSphere. SDDC on top of another SDDC.

K8s is concerned by its node load (which are vSphere VM), vSphere is concerned by its nodes load (ESXi hosts).
With this in mind k8s will move pods (containers) around its nodes, where vSphere will move k8s nodes around its hosts.

This indeed could lead to a spike or freeze of k8s nodes when both wish to move their loads at the same time. The solution is just an architecture solution: disable vSphere DRS on your k8s nodes so there won't be a move of your k8s nodes around while they are already moving containers.

Each time you overlay systems with the same behavior, you have to plan their definition properly to avoid race conditions.

Worst case would be running a docker swarm cluster on top of kubernetes on top of mesos on top of vSphere infrastructure on top of Nutanix partitions. (Absolutely doable but I don't think anyone would do that out of demonstration purposes)

So yes, this setup makes sense, it mainly require a careful design of all layers to achieve the best performances.


Include curl commands in a PostgreSQL Docker environment

One way to use curl inside the container will be to use a custom Dockerfile to build you postgrsql container.
Just use the image: postgres:12-alpine as the base image of your Dockerfile and install curl in alpine with the following command : RUN apk --no-cache add curl


What do I need to do as to prevent docker from downloading when I `run`?

Provided that the commands run on the same machine, yes, that should work.
To be even safer and to make it clearer, I would use an explicit version tag, such as 0.9.0beta1, or 0.8.0, instead of latest.


AWS Credentials - What are multi region user/role management best practices?

Start by reading this: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html

Those are best-practices and do not take into account business policies.

I don't think having one account per region is more secure than having policies scoped by regions. In cli, you will have to specify the region in the command anyway (not sure if AWS cli uses a default region if you don't specify one in your credentials file).


How to pass dynamic properties to an MSbuild task?

This question was kindly answered on StackOverflow. For anyone else looking for an answer:

<PropertyGroup>
  <BuildParameters>
    Configuration=Debug;
    Platform=Any CPU;
        SomeOtherProperty=Foo
  </BuildParameters>
</PropertyGroup>
…
<PropertyGroup>
  <!-- this property can even be extended afterwards, e.g. when a condition is needed -->
  <BuildParameters Condition=" '$(ShallAppendThings)' == 'true' ">
    $(BuildParameters);
    AnotherProperty=SomeValue
  </BuildParameters>
</PropertyGroup>
…

<MSBuild Projects="$(SolutionFile)" Targets="Build" Properties="$(BuildProperties)" />

source:
https://stackoverflow.com/questions/46226120/how-to-pass-dynamic-properties-to-an-msbuild-task


Salt Stack Reporting - Executing commands based on environment

The answer is that you can't directly. You have to set a grain on the minions first by doing something similar to:

salt '*' cmd.run 'grep environment /etc/salt/minion | awk "{print \"environment: \" \$2}" >> /etc/salt/grains'
salt '*' saltutil.sync_grains
salt -G environment:prod cmd.run 'df -h'


Azure DevOps/Pipelines seed job equivalent?

Yes this can be done, Azure Pipelines can ingest a yaml file, configuring the pipeline as code. These yaml files should be source controlled and the pipeline can be set up to build in an automated way.

You can also reuse steps within the pipeline


Technically mentioning DevOps might be wrong (the question you referenced is also subject to opinions).

But it can also be seen as clarifying/disambiguating - the term is widely used presently on the job market, not only according to your salary reference but also from the job ads themselves - see What is the difference between Sysadmin and DevOps Engineer?.

You'd only speak the recruiters' language - which is IMHO more important from the communication perspective than potentially being technically correct.

Language and culture are fluid - something wrong may become de-facto correct and vice-versa.


Is there a way through which I can control type of load balancer like round robin , least-connections etc using helm charts?

Well there are 2 possibble options.

One is to create a LoadBalancer type for each service you want to be exposed. In case you have just one service that's great. That's rarely the case of course :)

If you build your k8s cluster properly ( having Prometheus-Operator doing the monitoring for example ) most likely besides the business logic service/s, you would want to expose  internal cluster valuable services like Grafana.

That's why even if you have just 1 service, for future extension of such type of need the best would be to use an Ingress resource. It providees a way to expose multiple services through a single IP address —  HTTP level (network layer 7).

You could play with HTTP headers essentially and do some path/host routing.

You express the definition for your object and helm would do the installation. Another option is to use a tool like ArgoCD, basically sticking to GitOps policy.

A fundamnetal knowledge is to be aware that the Load Balancer/Ingress would forward the traffic to the service, not the pods. Services are abstracted entities by kube-proxy, by default kube-proxy uses a round-robin algorithm.

https://kubernetes.io/docs/concepts/services-networking/service/

Regarding the definition, it follows the standart K8s scheme [ apiVersion, kind, metadata, spec fields]. Example would be an Ingress created from AWS-Ingress-Controller.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: management
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
  - host: grafana.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 80
        path: /*


Store and analyize logs data from diffenrent application

Can you not use Kibana with Elasticsearch or does that not give you the 'advanced' features you need?

Have you looked at this connector for ES and Tableau? https://www.dremio.com/tutorials/unlocking-tableau-on-elasticsearch/


Kubernetes configuration with yaml anchors

YAML anchors are supported, but only for the same YAML file. You can't create the anchor (&) on a deployment file and reference (*) the value on another one.
If you want to share ENVs values across multiple Deployments, you can create a ConfigMap with the ENVs and use the envFrom spec. Example:
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-configmap
data:
  FIRST_ENV: my-value
  SECOND_ENV: another-value
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: beat
spec:
  selector:
    matchLabels:
      app: beat
  template:
    metadata:
      labels:
        app: beat
    spec:
      containers:
      - name: beat
        image: myimage
        command: ["celery", "-A", "wsgi.celery", "beat"]
        envFrom:
        - configMapRef:
            name: my-configmap
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flower
spec:
  selector:
    matchLabels:
      app: flower
  template:
    metadata:
      labels:
        app: flower
    spec:
      containers:
      - name: flower
        image: myimage
        command: ["celery", "flower", "-A", "wsgi.celery"]
        envFrom:
        - configMapRef:
            name: my-configmap


On-Site Octopus Deploy - How to deploy projects in a multi-project solution to different IIS locations

It can be achieved, however, you will make your life easier if you modify your TeamCity build process to create a package for each IIS Website then deploy each one individually using a Deploy to IIS step.

If it's impossible to change your build process then you will need to do some pre-processing or just deploy the websites through a custom script.


How to prohibit access to internals of Docker container?

In short, you cannot prevent your customers from modifying containers they run in their own infrastructure.  Containers are not like binaries that can be obfuscated; they are runtime environments.  The code you distribute inside the container, however, can be obfuscated.

Your question alludes to a third-party support issue: clients modifying software they run in their own environments.  If you provide tools to operate the containers you supply (e.g. monitoring and logging) then the clients should agree (as part of a software license) not to make unauthorized modifications to them.  This applies to all types of third-party software, not just containers.

Depending on your situation, you may also have the option to provide your app as Software As A Service (SaaS) running in a cloud infrastructure.

If your client requires your containers be run on their infrastructure, and refuses to abide by modification restrictions, then you probably don't want to try to support their use of your software.


As pointed out in the other comment, there are no constants in python. It's just a convention, but defining a variable in capitals, doesn't mean you can't reassign the value.

There are many ways to pass/define variables in ansible (command line, vars in playbook, include a yaml, vars dir in a role, defaults dir, etc) however, there isn't a way to define a constant.

The closest to setting a constant in the way you ask for it is setting your variables as defaults(that is defaults/main.yml inside the role structure). The defaults of a role are the type of variables that have the lowest precedence and are being overwritten by any other way you can define a var but are a great place to put a variable that is unlikely to change and won't clutter your "important" role vars.

Regarding the way you define them - I'd say it's up to you. Just stick to it. I haven't seen anybody using capitals and I would probably make the variable lower case, however, since ansible provides tremendeous amount of flexibility and freedom to do what you like the way you like it, it really is up to you.


According the documentation you're giving, you need to remove run-build from the command :
$ oc run NAME --image=<image> \
    [--generator=<resource>] \
    [--port=<port>] \
    [--replicas=<replicas>] \
    [--dry-run=<bool>] \
    [--overrides=<inline_json>] \
    [options]


Is a ReplicaSet pulling a fresh image at every spin up?

my understanding is that I'll probably need to modify the images pulled to do this.

Yes and no. It depends on how your ReplicaSet - specifically the associated Deployment - is configured. You can see the corresponding YAML configuration by doing

kubectl -n yournamespace get deployment yourappdeployment -o yaml
# or
kubectl -n yournamespace describe deployment yourappdeployment

Which will give you information about environment variables, images used, volumes, secrets, etc.

  I am trying to pull the images down locally so I can modify them

Pulling an image down and modify it in-place won't be of much help, except if you run a container based on this image and use something like docker commit yo actually update the image. Even if you do that, the Kubernetes objects may override aspects defined in your image (such as environment variables)

  is there a way to see how the ReplicaSet is pulling the images down?

Yes, first output the details of your ReplicaSet:

$ kubectl -n yournamespace get replicaset yourreplicaset -o yaml

kind: ReplicaSet
spec:
  ...
  template:
    ...
    spec:
      ...
      containers:
        ...
      imagePullSecrets:
      - name: some-secret-name

imagePullSecrets is the secret which should contains the credentials to pull your images. You can retrieve the data with:

$ kubectl -n yournamespace get secret some-secret-name -o yaml

kind: Secret
...
data:
  .dockerconfigjson: eyJh...VeryLongBase64String

This is Docker authentication data encoded in Base64. You must decode them:

$ echo "eyJh...VeryLongBase64String" | base64 -d

{"auths": {"registry.yourcompany.com": {"auth": "AnotherBase64EncodedString"}}}

Decode the auth value and you'll have your Docker registry login/password

$ echo "AnotherBase64EncodedString" | base64 -d

username:password


Is artifact promotion in this pipeline correct?

It depends on the artifact repository you are using but in general, you tag the artifacts in the repository to indicate their state.  This can specify it has passed some level of quality gate, some approval, or what ever is required to move through the stages.

I would say you tag the artifact as to it's state, not move between artifact repositories.  A single artifact repository can be used to hold the artifacts at multiple versions, and multiple states.  This provides a single audit point and a single way of controlling the deployment which can be very helpful.

With tagging of the artifacts you can be clear if they are ready for deployment, and if so to which environments.  The tags can be updated in multiple ways, but I would recommend automated processes, such as when an artifact is deployed, the artifact is tagged.  When it set of tests pass the end of that marks the artifact as ready.  If you have manual testing, that testing will be tracked in some tool or dashboard or approval process, when it is approved, that process should tag the artifact as ready to move on.


Since we are using ISTIO as Ingress Gateway with Kubernetes, the issue is related to ISTIO gateway config.
We simply had to replace the matching host to '*' in ISTIO gateway config


Can't access startfile http://localhost/

created with:
docker run -d -p 80:8080 -p 443:8443 jetty

gives a start page as:
Error 404 - Not Found.

   No context on this server matched or handled this request.

   Contexts known to this server are:

   Context Path Display Name Status LifeCycle
     __________________________________________________________________

   [1]icon  [2]Powered by Eclipse Jetty:// Server
     __________________________________________________________________

References

   1. http://eclipse.org/jetty
   2. http://eclipse.org/jetty


Is it possible to show a code coverage badge in bitbucket repositories?

You can certainly show a code coverage badge in a bitbucket repository, but it would have to be from an external source like jenkins, codeclimate, etc.

According to your link, if you are using pipelines they now show a code coverage badge on the project overview page. If you want to add that to your README or display it somewhere external, that functionality has not been implemented.


Terrafrom to declare both infra and application deployment in one state

I tend to prefer infra code living separately from application deployment code.  Multiple reasons for this:

You want your infra team to make infra changes.  You want your app team to make application changes (including application configuration).  Putting them separately lets you specify separate guard rails on them (for example, only the infra team is allowed to push changes to your infra repo).
You don't necessarily want to trigger a redeployment of the app if you make infra changes, and you rarely want to trigger a redeploy of your infrastructure if you trigger a new app build.
You don't want a bad build of your app preventing you from making any infrastructure changes until it's fixed, and vice-versa.

It actually makes sense to put your app deployment code (i.e. Helm chart) inside the app repo itself.
Then, you can actually package up application config changes along with the corresponding release of your application.  You also empower developers to do it themselves, without running the risk of breaking production more than they already can.
It will also allow your CI/CD tool to deploy both at once.
And since there's a strict mapping of app IaC and app itself in each GitHub branch, this actually lets you very easily create feature branch environments (i.e. a developer creates a new branch, your CI/CD picks it up, and automatically deploys it to a kube environment).


Execute multiple Ansible tasks with the same list of items

Make separate tasks file make_site.yml:

---
- name: Install apache site conf
  template:
    src: apache-sites-{{ site }}-conf.j2
    dest: /etc/apache2/sites-available/{{ site }}.conf
    mode: 0644

- name: Enable site apache conf
  command: a2ensite {{ site }}
  args:
    creates: /etc/apache2/sites-enabled/{{ site }}.conf

And in your playbook:

- include_tasks: make_site.yml
  with_items:
    - sitea
    - siteb
    - sitec
    - sited
  loop_control:
    loop_var: site


Docker swarm join linux container Error - remote CA does not match fingerprint

If you configured and joined correctly you should see two hosts: one for Linux and one for Windows.

Windows hosts cannot natively run inside the Linux hosts and vice-versa. This means you have to have different kernels for Linux hosts and Windows hosts in the same swarm. This way they can communicate across the same Docker network.

Refer to Docker Engine on Windows for more info.


How to store Jenkins job results and produce a summary report?

Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page):

... allows an easy and fully customized build report display.
Features

Allow a rich summary report visible from both project and build page
Reports must be written in an XML format according to the syntax described in section Syntax Description
This plugin allow the parsing of several XML files
Reports displayed are ordered according to the XML file names
Element displayed are based on

In general, the answer to such questions is usually found by searching the endless Jenkins plugin index.


Problem with Vagrant : Nothing happens on vagrant up command

If it's an Intel CPU, you may have to enable Intel VT-x in the bios. Sometimes the virtual machine will tell you this, and sometimes not. Here's a link that might help (ignore the Windows 10 part since you're on Win7). Also getting everything to the newest version would be a good idea if you can.


Kubernetes External Access Of A Single Pod

The NodePort will resolve correctly, no matter the node where the pods is (even if it gets rescheduled on a different node).

Any <Node IP>:<NodePort> will resolve to a Service type NodePort, no matter what node the pods are running.

Every node will forward traffic to that port to the Service.

You need to be on the same network (or create a SSH tunnel) or open that port in one of your nodes to the world.


VM Hosts: Single Large Host or Few Smaller Hosts?

A good rule of thumb is to try and avoid using one bit of logical kit (be it a single VM or a single physical host) for more than one thing.

If a physical server is going to be a VM host then that's all it should be. If a VM or a physical host is going to be a source code repository then that's all it should be etc. You can squeeze a bit extra out of your infrastructure if you consider everything to be a VM host and you then partition up those VMs to be single purpose hosts themselves.

The rationale behind this is that

A single host going down shouldn't impact your entire stack. Trust me this will hurt you badly as soon as you need downtime for an upgrade.
The cost to scale any individual component increases drastically if you want to try and scale on multiple similar servers (which will save you management dollars by reducing complexity) or it will cost you to scale out adhoc hardware as needed because you then need to deal with many different-sized hosts that are a pain to manage and require anyone managing them to know the platform intimately.

recommendation: split your infrastructure into as small a unit as is logical. If you can only afford to buy a single large bit of kit then make sure you split it into logically separate VMs or containers so that if you need to pop a bit of your infrastructure onto another server, or you need to scale stuff at different rates you can easily expand out but investing in a smaller upgrade.

Obviously the realities of software licencing, especially for windows OS's might make this cost prohibitive but, while I haven't caught up on windows licensing models in a while, I'm sure there are development centric licensing models that can suit your specific needs.


Bitbucket Pull Request Builder Configuration Jenkins

You can install 'Bitbucket Server Webhook to Jenkins' in Bitbucket.
https://marketplace.atlassian.com/plugins/com.nerdwin15.stash-stash-webhook-jenkins/server/overview

Once install you will need to enable the 'Post Receive' Hook on the required repository.

Remember to enable 'poll SCM' for the Jenkins Job.

Update
You can test the configuration of the plugin here:


What are the "User Mode" tools in kubernetes lens?

I have not used it, but the description in your image is understandable. It is adding RBAC (user permission) so that any user (also non-admins) can list namespaces (e.g. see all namespaces that exists within the cluster).
But it should not mean more than that, e.g. a user can not create a Deployment in a namespace he is not granted to.


Microservices terminology i.e. distinguishing between "core " and "other service instances"

The last couple of years I am/was part of couple of DevOps teams that are/were creating microservices. Some of the projects failed miserably.

Reasons:

architect that only read a blog about microservices and claims that he is the expert of microservices
team members that lack knowledge, but indicate that they know everything about it as the architect told them about it
lack of DevOps mindset
no support from management
thinking that microservices are the holy grail, i.e. Y=>X instead of X=>Y

According to me, a DevOps mindset is that one works together and that one focuses on deliverables, i.e. new software live within two weeks. If one wants to create all microservices in advance then a lot of assumptions have to be done and nothing will be put to production, i.e. available to the customer.

This happened in one of my projects. The architects was continuously talking about "what if this would happen and this" and so on. Outcome? After one year, nothing was running in production.

In my opinion, some companies start with microservices because they have heard about it, but do not know exactly what microservices are. Microservices are not the holy grail, e.g. one could create nanoservices, one should master automation.

My point of view about microservices is that one should start with a monolith first, e.g. if you want a webshop just start with it and ensures that it is running in production within two weeks. Secondly, start monitoring it, check if there are problems in certain areas, e.g. more bugs in component X or component Y is extremely slow. If it is clear that component Y is slow, create a plan of extracting it, rewrite it within two weeks, apply autoscaling.

I am in favor of the "cheese slicer method". As the cheese itself is a product that delivers value and is running in production and the X=>Y approach is used. Start with a problem and check what could solve this problem, instead of thinking about a solution, e.g. microservices (Y) that could solve a (hypothetical) problem.

For your information. I am currently busy with a personal microservices project that consists of two microservices, e.g. an api and a write microservice. I am first solving the impediments and cleaning up my backlog. Off course I have some ideas about next microservices, but I want to put first things first and add value to the customers instead of doing a lot without finishing this microservice project that I have already experienced a couple of times.

Note: once the current backlog is complete, an additional paragraph about

  If Microservices terminology i.e. distinguishing between core and
  other service instances

will follow.

April 26 2018

At the moment I have a monorepo that consists of 5 microservices, i.e. an API, writer, selector, insert and tableCreator. On one hand it is nice as the shared code could be used directly, but when a code change is pushed then I need to build everything.

The next steps is to create five repos and extract the code from the monorepo to each individual repo. Advantages of this is that a code change to a repo builds the microservice that has been changed, smaller code base (easier to read).

The common code will be moved to separate repos as well and published as artifacts so they could be used by the microservices.

One of the prerequisites is that CI and CD are enabled for each repo. Otherwise it will take a lot of time before everything has been deployed.

Microservices are not the holy grail

Automation is key

"Do one thing and do it well".


How to pass variables between build/deploy tasks in a TaskGroup in VSTS 16.131.28601.4

I'm not sure my approach is any "better". However, what I have done in the past was to use Table Storage or Cosmos DB to store the key-value pairs that needed to be persisted between each stage.

The general schema was:

Key: buildNum-variableName
Value: variableValue

I then created various wrappers for the SDK to access from PowerShell and MSBuild, this also enabled chaining multiple builds together to create a CI/CD pipeline.


How to set up AWS Trusted Advisor alert for Service Limit warnings?

That's on my backlog but still on early stage, I just surfaced the idea, so here's are my findings so far:

Using Trusted Advisor as a Web Service

Mainly:
Poll a Trusted Advisor Check for Status Changes

This documentation example can be implemented with aws cli with the support subset of commands and targetting the 'us-east-1' region, you'll then have to filter on the region field in the TrustedAdvisorResourceDetail structure part of the list.

I still didn't had time to write a script to poll the API (I would aim at ruby with the AWS SDK per our usage more than a shell script to group the requests).

Hope this help getting started even if that's not really a complete answer.

Courtesy of Romeo Ninov: For a  recap of which options is available for which account type you can check this site


Mastering CI infrastructure changes with GitLab

Gitlab CI supports running all of your builds inside Docker containers so you have a clean build environment every time. I'd recommend starting with the basic Docker executor but there is also an opportunity to autoscale your runners using Docker Machine or to execute the runners on a Kubernetes pod.

If you need to use Docker as part of your build (such as building, tagging and pushing images) then you can either use Docker in Docker or, preferably, bind mount the Docker socket of the host into your runner allowing it to run Docker containers as a sibling rather than child process. Note that the original author of DIND recommends using bind mounting instead of DIND, especially for CI.

To bind mount the Docker socket register your runner with the following command:

sudo gitlab-runner register -n \
  --url https://gitlab.com/ \
  --registration-token REGISTRATION_TOKEN \
  --executor docker \
  --description "My Docker Runner" \
  --docker-image "docker:latest" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

And then in your .gitlab-ci.yml file you can define your job as so:

image: docker:latest

before_script:
- docker info

build:
  stage: build
  script:
  - docker build -t my-docker-image .
  - docker run my-docker-image /script/to/run/tests

Using just the basic docker image rather than the docker:dind image


I'd say you can:

Using pip, install aws-cli for the www-data user specifically.
Example:

pip install awscli --user
Copying aws binary to some folder (for example awsonlybin) where it is the only command there and then adding the path of this folder to the www-data user's PATH. Example:

export PATH=$PATH:/usr/local/awsonlybin

In addition to that you should also do one of the following options:

Create an IAM user and access and secret keys and attach a relevant IAM policy to that user and then configure awscli to use that access key and configure them under the www-data user.
Create an IAM role and attach a relevant policy to that role and attach the role to that instance.


The vault docs mention a -field parameter for the read subcommand.  So you should be able to put this into a shell script:
SECRET=$(vault read -field foo secret/mysecret)

Other vault docs use the vault kv get in the same way so you might try:
SECRET=$(vault kv get -field foo secret/mysecret)


How to get into an App Container Manually with Garden-RunC Backend?

I suspect this may be something coming from containerd changes - running PCF 2.5 the runc list works fine, 2.6 runc list shows nothing in the inventory.  I have not yet found the equivalent command that will allow me to find the container id and how to execute a shell entry to that container instance yet, but I'm hot on the trail.


How to use tags in Amazon Web Services and Azure

I would set up a base taxonomy to start with, you can use automation across your estate to ensure that all resources follow the taxonomy.  You have a choice between simply deleting resources that don't comply or reporting on it to a central team who are responsible for chasing down and educating the owners.

I have followed the Best Practice set out by Microsoft for both Azure and Amazon:

Owner - normally this is a team distribution list although sometimes it is an individual if it is a "10% time" project.  To make sure that the e-mail address entered is valid we run a hourly scan of our estate to detect if there are new e-mail addresses in this field then send an e-mail with an activation link, failing to activate results in deletion after three days.
CostCenter - the numeric cost centre, we use this to feed into the end of month reconciliation.
SDLCPhase - one of Development, Test, SIT, UAT, PerfTest, PreProduction, Production.  We use this to define different automated behavour:

Development and Test: deleted overnight and at weekends to save on costs.  Teams are responsible for reprovisioning using the automation in the morning.
SIT, UAT and PerfTest: deleted when the cost center in our finance system is closed down.
PreProduction: scaled down overnight and when the cost center is closed down.
Production: we validate that all resources deployed to the production subscription have this tag, if they don't we send out e-mails extensively.

DropDeadDate - for us, this is a required field for everything except Production.  We run a nightly scan and send e-mails to the owner 7 days and 1 day before the DropDeadDate then delete the resource after it has passed.

In terms of the automation around this I have a tendency to use Janitor Monkey and some custom bots written is Azure Functions that sit in Slack Channels and do my bidding.


I was trying ti set up presto with azure gen2 , but facing the below issue when I am querying

Since ABFS mentioned in the stacktrace as a caching filesystem, then I suppose two possible reasons and suggest solutions for them

Dependency mesh in maven

As noted in two answers (one, two) to a similar question

  When we use maven-assembly-plugin, it merges all our JARs into one, and all META-INFO/services/org.apache.hadoop.fs.FileSystem overwrite each-other. Only one of these files remains (the last one that was added). In this case, the FileSystem list from hadoop-commons overwrites the list from hadoop-hdfs, so DistributedFileSystem was no longer declared.

So, check your dependencies and if there are JAR with correct org.apache.hadoop.fs.FileSystem in your distibution.

PrestoDB uses outdated azure-site.xml

There is known issue #11648.
And as said, ABFS, ADLS and ALDS gen2 was only introduced to https://github.com/prestosql/presto, not to prestodb

So, suggestion is to ask your question on Presto Community Slack

P.S. try storediag

There is a hadoop manual on Azure ABFS

And among other tips they recommend:

  One useful tool for debugging connectivity is the cloudstore storediag utility.

  This validates the classpath, the settings, then tries to work with the filesystem.


Is it possible to combine failure stage from groovy with try catch from pipeline?

You may use and update a flag to false in the catch and then use that flag in the when directive of other stage(s) that you want to skip because of any failure in the given stage. Example:

boolean flagStageOneSuccess = false

pipeline {
    agent any

    stages {
        stage ('stage-1') {
            steps {
                try {
                    // Do something that might fail

                    flagStageOneSuccess = true

                } catch (Exception e) {
                    // Handle the exception
                    flagStageOneSuccess = false
                }
            }
        }

        stage ('stage-2') {
            when {
                equals expected: true, actual: flagStageOneSuccess
            }
            steps {
                // Do something in stage-2
            }
        }
    }
}

It would also benefit you in viewing the status of your stage in Blue Ocean. If the stage is skipped, it shows as 'Not built'.


As per my understanding on cloud native infra.
Provisioning involves creating resources like VMs/instances, setting up IAM roles and policies, Firewalls, DBs, Load balancers, any clusters like EKS, GKE, etc.
And, configuration part involves, preparing your instances to run actually your apps by installing necessary softwares and packages, setting up tables in DBs and populating data, RBAC controls on resources, etc.


GitLab CI/CD page listing many gitlab runners that no longer exist

Just unregister offline runners:
As per GitLab Runner commands manual:

To delete the old and removed from GitLab runners, execute the following command.

    gitlab-runner verify --delete

You are not alone
P.S. The problem is common: there are many issues with questions like yours.
Moreover, there are several custom recipes to unregister "offline" runners:

bash script
API oneliner with curl
python script

API for "zombie" runners
As rlandster noted, sometimes you'll need an API
to unregister "zombie" runners.
curl -S --header "PRIVATE-TOKEN:<token>" "https://gitlab.example.com/api/v4/runners/all" | jq '.[] | select(.status == "offline") | .id' | xargs -I runner_id curl -S --request DELETE --header "PRIVATE-TOKEN:<token>" "https://gitlab.example.com/api/v4/runners/runner_id"`

This will list all runners, instead of the personal ones: https://docs.gitlab.com/ee/api/runners.html#list-all-runners
P.S. note about runners
Pls note that Gitlab itself doesn't manage runners. So, e.g. to restart or shutdown the runner, use appropriate commands on the runner's hosts.


What is Unit testing in the context of YML configuration Alerting rules?

The important part of the blog post you link to isn't testing the syntax of the YAML file; it's testing the semantics of the Prometheus configuration.

The core configuration in that post includes a fragment:

- alert: MyAlert
  expr: avg without(instance)(up) < 0.75
  for: 2m
  labels:
    severity: page

How do you know this is the right configuration?  If it's misconfigured then either your on-call team gets paged when the system is healthy, or it doesn't get paged when the system is down; both are bad.

The test setup shown in the blog post would let you test your Prometheus alerting setup, with artificial sample data (the "bar" instance is up for 5 minutes, then down for 5 minutes, and it's expected to page after it's been down for longer than 2 minutes), before you deploy it for real and without actually paging anyone.  That gives you some confidence your monitoring setup will do what you expect, before you deploy it for real.


How can I test native macOS UI software in a container before deployment?

I can confirm that running Docker on Mac works just fine, but according to this ServerFault question, it's not possible to just run MacOS in Docker.

(You might be able to run MacOS in a VM in a Docker container, but that's probably not what you want.)

You could purchase a MacMini, use it as a Jenkins slave and run automated tests on it as part of your build pipeline or something like that?


How to configure bitbucket pipeline and deploy to alibaba clouds

You can be done this task using below steps:

1: Creating the Bitbucket Git Repository And Adding our Application and Docker Configurations in it

2: Integrating Bitbucket with ACR

3: Deploying The Application

4: Securing our Application with Let’s Encrypt SSL/TLS

for full details check this:


What is the best strategy for running an ansible playbook within an ansible playbook?

If I've understood your problem correctly:

the ansible shell command in the middle could be converted to a simple playbook using the command task - let's call this middle.yml
your existing playbook could then directly import this middle.yml play (or even a more complex playbook).  In Ansible 2.4+, you would use import_playbook, specifically import_playbook: middle.yml, or in earlier versions, just include: middle.yml.

This processes the whole middle.yml playbook as if it was part of the main playbook - you have all the features of Ansible plays (hosts etc) and the output appears as part of main playbook.

This answer may need updating depending on your response to my comment above.


Cannot configure node group in new EKS cluster due to no Node IAM role found

This is actually pretty well described in the docs.

Before you create worker nodes, you must create an IAM role with the
following IAM policies:
AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy
AmazonEC2ContainerRegistryReadOnly

Also you need to make sure that this role could be assumed by EKS.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}


Assuming you are okay with moving your Jenkins(and settings) to a docker environment, you can do the following. This would give you the advantage to change ports(and manage jenkins) with greater ease in the future without ever having to touch IP tables or load balancers.

Host your Jenkins on a docker container and map the port 80 on your docker host daemon to 8080 on the container.

You can find the latest jenkins docker image here - https://hub.docker.com/_/jenkins

Install docker daemon and download the image using
docker pull jenkins

You can use docker run command to start jenkins along with the -p argument to supply port values.

For instance, you are hosting your docker daemon on 10.10.169.69, then you use this to start Jenkins
docker run -t container_tag -p 80:8080 image_name

And then to browse jenkins, use http://10.10.169.69:80/


Is there a website that shows the popularity of CI/CD tools?

I normally use stackshare. It doesn't show you usage as reported by the people running the tools but it has a decent community size and seems to be gaining more use rather than less.

It lets you compare tool features, community popularity and shows you who else is using those tools in their stack if that matters to you.


How I can use multiple hosts in AWX playbook?

I suspect you haven't set up the host in your inventory you can figure out where your hosts are being pulled from with the following command:

ansible-config dump | grep HOST

In practice, you probably just need to create an inventory file with the following content:

[nodes]
node01 ansible_host=IP OF YOUR FIRST NODE HERE!!!
node02 ansible_host=35.228.80.9

Then specify the -i parameter to ansible-playbook:

ansible-playbook -i inventory -k playbook.yml

Depending on your exact use cases it might be more scalable and maintainable to use GCE Dynamic Inventory, although this does require a little bit of extra setup.


How do I handle multiple AWS accounts in Terraform?

The answer is highly dependent on what you put inside it, if the states are only account related I'd go with a separate S2/dynamodb per account, if the states are cross-account then you'd need a central place to record them.


config.yaml is not setting up a new context

it seems you’re trying to change the config by setting a context and you’re not providing the config to the change context command. So do this
kubectl config --kubeconfig=infra_k8/config.yaml use-context test-sim

The setters are used when you want to add more entries to the config file, so you don’t need that command when changing context.


Conflating deployment environments and branches often leads to problems like you describe.  It does seem like a natural fit though and easily maps to mental model of an ideal solution but, real life is anything but ideal.

Basically, git is not a deployment tool, one should not clone to deploy a version but fetch a package (a tarball, zip file, docker container, NPM package etc).

In practice these two concepts are orthogonal.  Branching strategies (whichever it may be, gitflow, trunk based etc) is a means to assist developers isolating and sharing their work.  It ties naturally to the versioning strategy and is what forms the CI half of your CI/CD system.  Here you must force yourself to keep CI (continuous integration) separate from CD (continuous deployment) as you will see if you read-on.

Deployment however consumes the outcome of versions that were created and integrates various parts of the system and makes a whole that is usable, testable.

So if you view your branches and the code therein as a means to assemble a version which the build system will build and package and then store it somewhere.  Can be as simple as a network share or FTP site or it could tie in to an existing package manager with a appropriate back end, whatever suits your team best really.  This choice will depends greatly on the technology stack you are using and the underlying platforms you will be deploying to (static environments, docker, kubernetes, standalone copy-and-run etc).

Then another set of tasks can pull in any of these versions and assemble a working deployment.  Here this works best if your deployment environment are dynamic but it can be made to work with static environments as well.  This will form the CD part of your continuous system.

Once decoupled you can spin up an environment that would contain any versions that has been previously built.

In term you could even have multiple production environments with slight variations of a user interface tweaks and pit them against each-other with techniques like A/B testing and clever load balancing / distribution to end-users.

Going back to your particular situation it will simplify your gitflow back to a more traditional master/develop permanent branches and a set of features/hotfix and release branches.

Using semver you can keep official versions strictly for master merges (from release branches or hotfix branches) and pre-release versions for all other branches.  for your pre-release branches simply add a -SOMETHING where something is the name of the branch for example and possibly a timestamp to differentiate successive deployments.

for example,

release branches would get the x.y.0-RCi (i th release candidate). x.y is the version that will be released where the -RC is simply dropped when merged in master.  0 as the patch because release branches are used for new features and product evolution
hotfix branch follow the same principle as the release branch except the patch number is set to a non-zero value.
development branch could get x.y.z-dev.timestamp
feature branch x.y.z-JiraID-timestamp.  Where JiraID is the ticket number of the system you use to track new features (like Jira, Bugzilla, etc)

I propose semantic versioning here because I am familiar with it but the same broad principle of segregating branches through versioning can be applied with pretty much any versioning scheme.

Nowhere in the source control do you see anything that relates to deployment environment.  Only your versioning strategy is what links them, they are otherwise completely isolated and independent.

Your devs will feel more comfortable in this new environment and your deployments will feel less constrained and awkward.

Hope this helps


How is Ansible different from simply running a provisioning bash shell in Vagrant?

I never used Ansible but since a few weeks, I try to figure out what good Ansible could be in comparison with shell scrips–Which proves, at least in my case, that the haunting ad-campaigns they run are effective! After many unsuccessful attempts–which proves how their documentation fail at answering one of the most obvious question–I think I finally got it.

My conclusion is that over shell scripting, Ansible essentially offers 1. The possibility of checking that a system agrees with a desired state, 2. the ability to integrate with Ansible Tower, which is a paying system that seems to include monitoring abilities. In some important cases, like when implementing the immutable server pattern, the point 1 is probably not very useful, so the list of advantages is rather thin.

It seems to me that the benefits offered by Ansible over shell-scripting, as the documentation present them, could be sensible in a few handful of optimistic cases well covered by available modules but are small or even hypothetical in the general case. For a skilled shell-programmer, these benefits are most likely counter-balanced by other aspects of the trade-off.

But my conclusion maybe only proves how bad the introduction material is at displaying the actual advantages of this software!

Now, I propose to watch the introduction video and go randomly as a potential new user through the introduction material to Ansible an let's compare it to what a skilled shell programmer can produce in a reasonable time.

The quick start video:

There is a quick start video. It starts with a page claiming that… well these are not really claims, these are bullet lists, an artefact commonly used to suspend critical judgement in presentations (since the logic is not shown, it cannot be criticised!)

1. Ansible is simple:

1.1 Human readable automation – Specifications are technical documents, how could

  name: upgrade all packages
  yum:
    name: '*'
    state: latest

be easier to read than the corresponding yum invocation found in a shell-script? Furthermore, anybody who had contact to AppleScript dies laughing when they read “human readable automation.”

1.2 No special coding skills required – What is coding if not writing formal specifications? Ansible has conditionals, variables, so, how is it not coding? And why would I need something I cannot program, that would henceforth be inflexible? The statement is happily inaccurate!

1.3 Tasks executed in order – Well, maybe some codegolf aficionados are aware of languages that execute tasks in disorder, but executing tasks in order hardly looks exceptional.

1.4 Get productive quickly – Skilled shell programmers are productive now. This counter-argument is just as serious as the initial argument.

2. Ansible is powerful

A popular salesman trick to sell artefacts is to fool people into believing they will acquire the “power” of these artefacts. The history of advertisement for cars or isotonic drinks should supply a convincing list of examples.

Here Ansible can do “app deployment” – but shell script surely do, “configuration management” but this is a mere statement of the purpose of the tool, not a feature, and “workflow orchestration” which looks a bit pretentious but no example in this document goes beyond what GNU Parallel can do.

3. Ansible is agentless

To populate the column, they wrote in three different manners that this only needs ssh, which, as everybody knows is a daemon and has nothing to do with these agents pervading the world of configuration management!

The rest of the video

The rest of the video introduces inventories, which are static lists of resources (like servers) and demonstrates how to deploy Apache on three servers simultaneously.  This really does not match the way I work, where resources are highly dynamic and can be enumerated by command-line tooling provided by my cloud provider, and consumed by my shell functions using the pipe | operator.  Also, I do not deploy Apache on three servers simultaneously, rather, I build a master instance image that I then use to start 3 instances which are exact replicas one of the other. So the “orchestrating” part of the argumentation does not look very pertinent.

Random documentation step 1: Integration with EC2

EC2 is the computing service from Amazon, interacting with it is supported by some Ansible module. (Other popular cloud computing providers are also provided.):

# demo_setup.yml

- hosts: localhost
  connection: local
  gather_facts: False

  tasks:

    - name: Provision a set of instances
      ec2:
         key_name: my_key
         group: test
         instance_type: t2.micro
         image: "{{ ami_id }}"
         wait: true
         exact_count: 5
         count_tag:
            Name: Demo
         instance_tags:
            Name: Demo
      register: ec2

The corresponding shell-script would be essentially identical with YAML replaced by JSON:

provision_a_set_of_instances()
{
  aws --output=text ec2 run-instances --image-id …
}

or the JSON version

provision_a_set_of_instances()
{
  aws --output=text ec2 run-instances --cli-input-json "$(provision_a_set_of_instances__json)"
}

provision_a_set_of_instances__json()
{
  cat <<EOF
{
    "ImageId": …
}
EOF
}

Both version are essentially identical, the bulk of the payload is the enumeration of the initialisation values in a YAML or JSON structures.

Random documentation step 2: Continuous Delivery and Rolling Upgrades

The largest part of this guide does not display any really interesting feature: it introduces variables (IIRC, shell scripts also have variables)!, and an Ansible module that handles mysql, so that if instead of searching after “how do I create a mysql user with privileges on X Y” and end with something like

create_application_db_user()
{
  mysql --host "${mysql_host}" --user "${mysql_user}" --password "${mysql_password}" "${mysql_table}" <<EOF
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%';
EOF
}

you search after “how do I create a mysql user with privileges on X Y in ansible”  and end up with

- name: Create Application DB User
  mysql_user: name={{ dbuser }} password={{ upassword }}
              priv=*.*:ALL host='%' state=present

The difference is still probably not very meaningful. On that page we also discover that Ansible has a template meta-Programming language

{% for host in groups['monitoring'] %}
-A INPUT -p tcp -s {{ hostvars[host].ansible_default_ipv4.address }} --dport 5666 -j ACCEPT
{% endfor %}

When I see this, I happen to really be in my comfort zone. This kind of simple meta-programming for declarative languages is exactly the same theoretical paradigm as BSD Makefiles! Which I happen to have programmed extensively This excerpt shows us that the promise of working with YAML file is broken (so I cannot run my playbooks through a YAML parser, e.g.). It also shows us that Ansible must discuss the subtle art of evaluation order: we have to decide if variables are expanded at the “declarative part” of the language or at the “imperative” meta-part of the language. Here shell programming is simpler, there is no meta-programming, aside from explicit eval or external-script sourcing.  The hypothetical equivalent shell excerpt would be

enumerate_group 'monitoring' | {
  while read host; do
    …
  done
}

whose complexity in comparison to the Ansible variant is probably tolerable: it just uses the plain, regular, boring constructs from the language.

Random documentation step 3: Testing strategies

Last, we meet what turns out to be the first actually interesting feature of Ansible: “Ansible resources are models of desired-state. As such, it should not be necessary to test that services are started, packages are installed, or other such things. Ansible is the system that will ensure these things are declaratively true. Instead, assert these things in your playbooks.” Now it starts to be a bit interesting, but:

Aside from a handful of standard situations readily implemented by available modules, I will have to feed the bits implementing the test myself, which will quite probably involve some shell commands.
Checking for the conformity of installations might not be very relevant in the context where the immutable server pattern is implemented: where all systems running are typically spawned from a master image (instance image or docker image for instance) and never updated – they are replaced by new instead.

Unaddressed concern: the maintainability

The introductory material from Ansible ignores the question of the maintainability.  With essentially no type system, shell-scripting has the maintainability ease of JavaScript, Lisp or Python: extensive refactorings can only be achieved successfully with the help of an extensive automated testsuite – or at least designs that allows easy interactive testing. That said, while shell scripting is the lingua franca from system configuration and maintenance, nearly each programming language has an interface to the shell. It is therefore totally feasible to leverage the maintainability advantage of advanced languages, by using them to glue together the various the bits of shell-configuration bits. For OCaml, I wrote Rashell that essentially provides a hand of common interaction patterns for subprocesses, which makes the translation of configuration scripts to OCaml essentially trivial.

On the side from Ansible, the very weak structure of playbooks and the presence of a meta-programming feature make the situation essentially as bad as it is for shell scripting, with the minus points that it is not obvious how to write unit tests for Ansible, and the argument of introducing ad-hoc a higher-level language cannot be mimiced.

Idempotency of configuration steps

The documentation of Ansible draws the attention on the necessity of writing idempotent configuration steps. More precisely, configuration steps should be written so that the step sequence a b a can be simplified to a b, i.e. we do not need to repeat configuration step. This is a stronger condition than idempotency. Since Ansible allows playbooks to use arbitrary shell commands, Ansible itself is unable to guarantee that this stronger condition is respected. This only relies on the programmer's discipline and the importance of this variation of idempotency when writing configuration scripts is certainly not a novelty.

Post-Scriptum. Since this answer seems to enjoy a relative popularity, I fixed a few embarrassing syntax errors and typos. By a twist of life I also had to use Ansible two years in my work. Overall my experience confirms what I foreseen here and I hardly can think about a situation where shell scripts would have been really outperformed by Ansible. On some aspects, Ansible is just worse than shell scripting. At least the shell has functions, these functions can be mocked, it is possible to test part of all of them, so overall the shell has much better software engineering features than Ansible has. In a shell script it is also possible to process data and awk can express all what SQL can, which is very important when programming configuration – the information we are working with here is not intrinsically hierarchical, so there is a need for extracting an rewriting. Ansible is so bad at extracting and rewriting data! Treatments must be expressed with a mixture of YAMl-templating at the playbook step level and a dialect of Jinja at the dictionary member level… this is cumbersome, ugly, hard to write, hard to test and poorly documented (I regularly looked up the Jinja filter implementations!).


If what you are looking for is a chef cookbook to manage MySQL, I would recommend checking this It is an officially maintained cookbook. Keep in mind that:

  This cookbook is concerned with the "MySQL Community Server", particularly those shipped with F/OSS Unix and Linux distributions. It does not address forks or value-added repackaged MySQL distributions like MariaDB or Percona.

I hope this could help.


Is it possible to autodeploy on every commit in a specific branch using Travis CI and GitHub?

Instead of using Travis' deployment features, build the deployment logic in to your normal "test" script(s). For a PaaS this might be as simple as pushing to another git remote if all tests pass, but the sky is the limit.


Helm is a Kubernetes tool so this answer assumes you are deploying to Kubernetes or OpenShift not Docker Swarm.

If you are using Helm well then I don't think it is particularly important what build pipeline tech you are using to invoke it. I think what is more important is how you manage all your configuration within Git and how you structure the Helm deployments.

There are two different paradigms you can follow:

Build the app container image with the correct environment variables baked within it for each environment.
Build only one app container image that picks up environment variables from within the environment it has been deployed.

If you follow 12factor.net best practices then you go for Option 2. IMHO it feels far cleaner and less error-prone to move the exact same image between kubernetes namespaces that represent different environments. We store environment variables inside of Kubernetes as secrets or configmaps. We then get to two more questions:

How do you load secrets and configmaps as environment variables into your .Net Core runtime images when the container starts up?
How to you manage the secrets and configmaps in each environment?

With respect to the first question, Kubernetes has this covered. Any deployment can name a bunch of different secrets that should be loaded as environment variables. Ditto for configmaps. You just need each environment to be a different namespace so that the same logical secret can have the same name in each environment. Your database credentials can then be a secret called main-database-credentials that holds different credentials in different environments.

To deal with the second question we use Helm to manage both secrets and configmaps. We have one generic Helm chart that deploys all our different secrets in all our different environment:

helm upgrade --install -f $NAME.yaml $NAME secret-1.1.tar.gz

So to upgrade our main-database-credentials secret we would change the settings in main-database-credentials.yaml then tell the chart to recreate the main-database-credentials deployment using that command. The chart simply uses the Helm templating language to build yaml that defines the secret using the name set in the yaml file. Likewise, we have a generic chart that updates configmaps that is invoked with a very simlar command as above. Helm uses a standard Golang templating library so it is very straight-forward to create those generic charts.

At first, it may seem a bit heavyweight to use Helm to update something like a database password secret. Yet helm gives you a deployment history and lets you diff the history of deployments. It also lets you easily roll-back to a previous deployment. That is really powerful as an audit trail of config changes that is really helpful when managing a complex microservices deployment. Since we also use Helm to deploy all our apps it makes perfect sense to use the same power tool to manage the deployment of configuration into our environments.

As we can use a generic helm chart to deploy many secrets into many environments why not use a generic helm chart to deploy all our microservices? We do:

helm upgrade --install -f $NAME.yaml $NAME deployer-1.3.tar.gz

This means that NAME=my-microservice and NAME=my-admin-webapp or anything else can be deployed using one generic deployer chart. The path and tag of the application container image to deploy is in the yaml file. The container path is different for different apps and the container tag can be different between environments.

That might sound strange as normally Helm packages a particular app so you would expect a one-to-one relationship between charts and apps. Yet if all your microservices are containers following 12factor.net best practices shouldn't they all "look the same" from the perspective of deploying them; it is just the container and the environment variables you deploy into the one generic chart. I see it that the app is the pure container with none of the configurations to run it and the Kubernetes configuration to run it is "application infrastructure". So I have a Helm chart for each type of application infrastructure. I don't want each bit of application infrastructure to be hand-rolled I want to cookie cutter create it using Helm.

Now we have covered all the Helm mechanics we are left with the question "how do we manage all our configuration within git?". We use different repo for different purposes:

Each microservice has its own repo. We use CircleCI to build the code into an image that is configured via environment variables. We don't store any Helm or Kubernetes config in this repo. We do have a properties file that names the latest version of the deployer chart this code is compatible with.
We have a GitHub repo that holds the source of the generic Helm charts that are used to deploy all our apps and secrets. These don't actually change very often now they are fairly stable. CircleCI builds and releases the charts as tar.gz onto the GitHub releases page for this repo.
We have a git repo per environment. This has a folder per deployed microservice within that environment. Each contains the yaml file to pass to the helm deploy encrypted with git secret. It has a folder per deployed secret within that environment. Each contains a yaml file that is git secret encrypted. Ditto for configmaps.

Why a repo per environment? A branch per environment sounds more normal. Only a database credential for the live environment never moves between environments. As we use a separate repo for live we can give it separate permissions. We might let all developers go wild in the dev-environment repo we have a bit more discipline with the live-environment repo.

We then have CircleCI do automated deployments of configuration. This works as Helm data detects changes and doesn't deploy a change to any Kubrernetes object if the latest yaml it generates is already deployed. To change a password in live you change it in the git secret encrypted yaml in the live repo and apply a release tag. CircleCI pattern matches on the tag name to see that its a release tag and it runs the deploy step. That uses a container with the helm client to run a bash script that loops over all app folders, secret folders and configmap folders running Helm for the right chart and correct yaml. Helm will only apply updates where there are changes within git.


How to solve "E405: Remote package tasks should have a retry" in Ansible?

ansible-lint warning 405 can be quite irritating as in certain cases, i.e. when running locally retrying is very unlikely to make any difference.  Actually resolving it is normally simply a case of adding the following:

register: task_result
until: task_result is success
retries: 10
delay: 2

When this is added to task Ansible will retry ten times with a delay of two seconds between each retry until the returncode is 0.


How to create a docker registry secret without passing the password on the command line?

It is possible to create a secret without passing on the commandline by reading it from file:

$ kubectl create secret generic ssh-key-secret \
  --from-file=ssh-privatekey=/path/to/.ssh/id_rsa \
  --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub

See https://kubernetes.io/docs/concepts/configuration/secret/ for more information.


What's the relationship between ContainerD and Docker?

Containerd is a daemon for running containers.  The Docker blog has a nice write up explaining how this fits into Docker.  While containerd is used in Docker, it can also be used as a daemon for other container solutions.  Containerd requires runc in order to function.

runc as defined by the project is a "CLI tool for spawning and running containers according to the OCI specification".

These tools are designed to follow the Open Container Initiative (OCI) specifications.  This has become a standard and defines how Docker and other tooling interacts with containerd and runc.

When you look at most Kubernetes diagrams they will not show containerd or runc as they are abstracted away into the container engine you use (e.g., Docker, rkt).  Because Kubernetes uses containerd, it will also use runc.  This is what allows multiple container solutions that follow OCI specifications to integrate with Kubernetes.

So yes, you could break down how part of Kubernetes is built by looking at runc, containerd, and the container engine.


Devops relate to the execution of software. It is all about how to run an application or service. So server configuration tools, virtual machines and containers are the next steps. I suggest looking at the documentation of those tools and learning from them things like discovery, observability, scalability and redundancy. Just to name some examples:

Server config:

PXE
Ansible
Puppet

VM:

Vagrant
QEMU

Container:

Docker
Kubernetes

EDIT: Cloud technologies overview:

CNCF landscape

When your software architecture is about microservices, then containers are a good choice. When you are using old monolithic applications then VMs are not a bad option. So it always depends on the context what you need to learn specifically.
The topic is rather broad to summarize it. A book might help with that but sadly I cannot suggest a good broad one. "Kubernetes in Action" gives you a good start in Kubernetes though.


Cloud DB operator responsibilities / common tasks

I always saw a DBA as someone who fits in between developers and operations when it comes to database management.

On the one hand, they often take care of backups, clustering, replication, actual binary installation, file system management, etc.

On the other, they also take care of relational DB schema, do performance or query optimization, and advise developers on the best way to lay out their application storage and configuration schemas.

In theory, the first job can be done by a system administrator.  The second job can be done by a senior developer or architect.

In practice, it often makes sense to have someone who's well versed in all parts of running a database.

Provisioning the instance is only a small part of what they do.  Having your instance hosted in the cloud doesn't remove the need to have someone manage your schema or optimize performance.  A cloud RDS with default settings likely won't be aware that key_buffer_size value needs to be set to 16M or some of your application's features won't run.


What's the way to deploy a resource once on a helm chart that is deployed multiple times?

I've eventually created an umbrella chart with alias:

dependencies:
  - name: app
    repository: file://./app
    version: 1.0.0
    alias: child1
  - name: app
    repository: file://./app
    version: 1.0.0
    alias: child2

And in values.yaml i've changed the needed values,
then added the grafana dashboard in the bundled yaml file.


Are there any best practices and/or tools for cleaning old Docker images on a regular basis?

I've had good luck with
Spotify/docker-gc. It can be run from cron or as a docker container.

  A simple Docker container and image garbage collection script.

  Containers that exited more than an hour ago are removed.
  Images that don't belong to any remaining container after that are removed.


Compound calculation of GCP GCE SLAs based on number of instances?

99.99% SLA uptime roughly translates to 52.5 minutes a year of downtime (calculator here).  Generally if the service is down, it is down for everything in that geographical area.  So if you had 1000 servers in the same geographical area and that area goes out of service, your downtime would be tied together.

However, most of the "big" cloud providers (e.g. Google) provide their services across different regions and zones.  By spreading your infrastructure out across different areas, you can make your network more resilient to cloud provider outages.


Few ideas to help you

When you define a "CodeBuild" project, you can specify which base image it will use to execute your commands. The one you are using is not Ubuntu yet you are trying apt-get. Try a different base image.
Provide us with a better log stream.

Exporting Logs from CloudWatch

I found that the easiest way is to export logs through command-line:

click on 'View Logs' link, to find out your Log Group and Log Stream
install "awscli" and "jq"

Run substituting your log-group-name and -log-stream-name:

aws logs get-log-events --log-group-name /aws/codebuild/Dumper  \
    --log-stream-name 0bd74b8c-74b9-4f9b-b275-167b45901aa3 | \
    jq '.events[].message' -r | grep -v '^$'

This should give you the output, which you can include in the question.


Jenkins - Active Choice DynamicReferenceParameter not returning list

I figured out what was going on. The return from hosts var was a single string with comma separated hostnames host1, host2, I had to convert it to List  using the comma as delimiter. The following post How to convert comma-separated String to List? helped me with that task.
Basically the last section now reads
                script:
                """
                List<String> list = Arrays.asList(hosts.split("\\\\s*,\\\\s*"));
                return list
                """


How to setup Docker mirror proxy Repository in Nexus Repository ManagerOSS 3.19.1-01?

In Nexus, go to the Admin Backend.

Create a blob store if you haven't already. Guidance can be found here: https://help.sonatype.com/repomanager3/configuration/repository-management#RepositoryManagement-BlobStores

Once you have a blob store, continue like so:

Choose "Repositories", hit "Create Repository". Choose "docker (proxy)" as the recipe.

Give it any name (for example "docker-hub").

Under "Remote storage" enter https://registry-1.docker.io.

As "Docker Index" select Use Docker Hub.

Select a blob store for it (which you have created above).

Hit "Create repository". No more configuration is needed for the proxy repo.

Next up: A group repository, from which you will pull:

Go back to "Repositories" and hit "Create repository" again. This time select "docker (group)".

Call it anything you like (for example "docker-group").

For "HTTPS" enter any port you like. (we have 10500)

For blob store select any blob store (for example the one you have created above).

Under "Group" select for "member repositories" the repository you have created above.

You can also add a "docker (hosted)" repository as well if you have one set up. (I will explain later)

Hit "Create repository".

Now open your Docker cli and do a pull of some Docker Hub image but don't pull it from Docker Hub, pull it through the HTTPs endpoint of your docker (group) repo that you have created above like so:

$ docker pull my.repository.com:10500/phpmyadmin/phpmyadmin

This will create a pull request to your Nexus OSS, which will proxy the request to Docker Hub. The image from Docker Hub will be cached in your Nexus and will be delivered to you.

After pulling browse your docker (proxy) repo that you have created above and check the images inside. You will find a phpmyadmin/phpmyadmin inside (or any other image you have checked out) with a current timestamp.

Hope that helps!


I just configured promoted builds plugin and the pipelines plugin and found at least one difference between the two plugins, i.e. the first plugin seems to be able to allow manually deploy of certain stages, while the pipelines plugins seems to deploy everything automatically.

According to this answer, manual deploy of a certain stage seems to be missing.

In summary, it seems that it would be nice to combine the two plugins as it could be handy to have a manual check before deploying to production, i.e. a human has to press the button in order to deploy to production.

According to this answer it is not possible to combine both plugins at the moment.


From 6 months to 16 seconds? grokking DevOps speedup data from DevOps report by DORA and Puppet Labs

I wouldn't exactly calculate it like that. It doesn't necessarily mean that the actual verification time for a changeset is decreasing that much.

The verification process itself can be (and often is these days) a processing pipeline in which the average/effective verification time per changeset can, indeed, decrease to insanely small amounts.

For example the processing pipeline can process multiple changesets simultaneouly. If you have, let's say, 60 proposed changes, each one of them touching a different file in a repository, you can combine them into an equivalent single uber-changeset touching 60 files. You could then process this uber-changeset through a verification step that takes, let's say, 1h. If the verification passes you obtain a speedup factor of x60 and an effective verification time per changeset of 1 min, even if the actual verification time was 1h. Granted, you'll lose some of the speedup when the verification fails and you need to perform bisections to identify the changeset at fault, but that's a different story.

Or you can have each of the changesets verified independently, through 60 1h verifications performed in parallel. Same effective verification time, but with a whole lot higher verification resource usage.

The point, I guess, is that there are ways of building processing pipelines which can greatly reduce the effective per-changeset verification times and still keep very large scale projects agile.

Another point to consider is the branching strategy. Those 6 months weren't the actual verification time. Most of that time was simply spent navigating through the branch spagetti and fighting the integration hell, just to propagate the fix into the proper shipping branch - effects of the waterfall development model which was considered the norm at the time.

Today CI/CD (the proper one, not the CI Theatre) and Trunk Based Development eliminate all that wasted time/effort and side-stepping, with the fix being virtually ready for delivery right from the moment it is integrated in the trunk - no unpredictable massive branch merging obstacles ever stand ahead of shipping.


What is the definition of "cattle not pets"?

Randy Bias chronicles the history of the term stating that it probably originated in 2011 or 2012 when Bill Baker used the analogy when describing "scale-up" vs. "scale-out" architectural strategies. Bias adopted this into his presentations about cloud architectural patterns:

  In the old way of doing things, we treat our servers like pets, for example Bob the mail server. If Bob goes down, it’s all hands on deck. The CEO can’t get his email and it’s the end of the world. In the new way, servers are numbered, like cattle in a herd. For example, www001 to www100. When one server goes down, it’s taken out back, shot, and replaced on the line.

Bias continues to define Pets as

  Servers or server pairs that are treated as indispensable or unique systems that can never be down. Typically they are manually built, managed, and “hand fed”. Examples include mainframes, solitary servers, HA loadbalancers/firewalls (active/active or active/passive), database systems designed as master/slave (active/passive), and so on.

and cattle as

  Arrays of more than two servers, that are built using automated tools, and are designed for failure, where no one, two, or even three servers are irreplaceable. Typically, during failure events no human intervention is required as the array exhibits attributes of “routing around failures” by restarting failed servers or replicating data through strategies like triple replication or erasure coding. Examples include web server arrays, multi-master datastores such as Cassandra clusters, multiple racks of gear put together in clusters, and just about anything that is load-balanced and multi-master.

Fundamentally, what Bias and Baker are trying to convey is there has to be a transition from how we treat servers from being "Unique Snowflakes" with names and emotional attachments, to a model whereby if we have a problem with the server we create a replacement and destroy the problematic server.

Finally, it is probably worth mentioning that in regulated environments taking a server out the back and shooting it may not be optimal.  In these cases it is often advantageous to "freeze" the server, for example using docker pause to freeze a container.  This can then be used to perform a Root Cause Analysis as part of the Incident or Problem Management Process.


how to prevent Ansible from showing output on screen?

This can be done by judicious use of one of the Ansible callback plugins. Set the relevant configuration in your ansible.cfg file.

You have a few options...

In order to print nothing to the screen, you can use the null callback:

  This callback prevents outputing events to screen

However, this may be overkill and doesn't do what you want, which is to write the output to a file. This can be done with either of the

tree or
log_plays

modules.

I would probably suggest the log_plays module in your case.

Note: This is going straight from the documentation. I haven't used the tree or log_plays plugin myself, opting either for the yaml or profile_tasks one, to make things more readable. It also seems like you want to filter just a part of the output, not all of it. You may want to use a combination of filters in order to get the desired effect.


Integrate iOS Xcode Build Node with Jenkins X + Tekton

I don't think OS X builds can be done with Tekton yet. For builds that need OS X, I would currently consider to use GitHub Actions with support for macOS actions.


Well, you can pass multiple goals to maven in command line simply by passing them as arguments.
Skipping test require a java system property that you can also pass as an argument to the command line.

For packaging a maven artifact, executing a sonar analysis with the maven goal but skipping executing the tests you should use :

mvn package sonar:sonar -DskipTests

Before that, you need to decide if you want to put sonar settings in the maven configuration (i.e. pom.xml or .m2/settings.xml) or in the Jenkins server configuration.

With maven only, the sonar:sonar goal needs some requirements and maven configuration that you can find here in the documentation. Jenkins use will be reduced to execute maven, you won't need to install a Jenkins plugin.
Using Jenkins with sonar also needs some requirements and Jenkins configuration that you can find here in the documentation. Be sure to install the Sonarqube scanner Jenkins plugin. This plugin enable some Jenkins integration features.


How can I convert a production Linux server into a docker Image or container?

Is that possible without any hack?

Yes, it is possible. While Docker is often used for microservice-oriented images (i.e., one single service per image), nothing keeps you from converting a monolithic old application to Docker by building a large image, and running arbitrarily many processes inside.

While it is true that a Docker image is not a standalone VM, it still contains everything an equivalent VM would contain - excluding the kernel. I.e., all OS libraries (libc, everything else...) are in there anyways.

  If yes, what are the big lines to follow?

A Dockerfile is just a funky version of a plain old shell script which you could ordinarily use to provision your software with all its dependencies. That is, you chain a lot of commands, which are executed during the build process, and whatever ends up in the file system at the end gets packed up into an image.

So the general route is:

Pick a base image. Depending on your needs and security requirements, you can pick your favourite distribution from Dockerhub. At first, you can pick any Debian, Ubuntu or whatever distribution you prefer and are used to. See below for further considerations.
Those base images usually are pretty small. Hence, your next step is to use RUN apt install... (or whatever package manager your base image uses) to add any standard packages you like.
Finally, you have to install your own software somehow. If you follow the usual approach, you would prepare any binaries and other files you need in the host-side directory alongside your Dockerfile, and then ADD them in.
The last step is to (optionally) create a ENTRYPOINT, i.e., a command that is run when the image is started up, as well as EXPOSE routing information, run as non-root USER or set ENVironment variables.

These are the big lines. If you just do it naively, you will end up with a huge image (gigabytes), and create a new huge image everytime you rebuild, wich can pose serious problems. The next steps are thus:

Switch the base image to a smaller one. alpine is good.
For each RUNcall, make sure there are no temporary/unneccessary files left afterwards. For example, apt install leaves you with a lot of cache junk in /var (same for the other package managers). Due to the way Docker works with its onion-style layered file systems, you have to get rid of the junk within the same RUN command, you cannot clean up in a second RUN. If you google, you will find plenty of examples.
You probably pulled in too many packages with your package manager, fix that.
For installing your own software, the same applies. Avoid having temporary .tar archives lying around which you don't need anymore.

These steps should significantly reduce your image size already. Depending on what you are actually doing, 500'ish MB is a good target to avoid over-optimizing at this stage.

The final step is to optimize everything:

Make sure that you do not have to rebuild every single intermediate image each time, by ordering the lines in the Dockerfile appropriately. You can get away with large images if the "largeness" happens in the first few image layers, if those change seldom, and can be re-used by many final images.
For an intermediate image to be re-used, it must be bitwise identical to another one. So be sure to fix all variable information (like version numbers etc.) as best you can.
Obviously, if you get the hang of all of this, and get your complex, many-part application running inside a single Docker image, then it's time to split it apart across multiple images (rinse, repeat). Anything that communicates by TCP/IP should be an easy choice to do so.


when does currentBuild.changeSets in Jenkins gets filled?

It will be set after any checkout steps (or derivatives of checkout such as git or svn).

So for instance:

println(currentBuild.changeSets) // should print an empty set

checkout(scm)

println(currentBuild.changeSets) // should print out any changes in the current build


Docker calls to detect app crashing inside it

Is there a way to know if an app has crashed inside docker

If your app is writing logs and is the main process in that container, then yes, just watch them

  am implementing CI/CD and want to stop container when it crashes on auto-deploy

What's your tools used for CI/CD? For example, when you are releasing an app with kubectl apply you can wait till automated deployment succeed: kubectl rollout status


How to connect to host during a Docker build?

From your container's perspective localhost is the container itself.  If you want to communicate with the host machine, you will need the IP address of the host itself.  Depending on your Operating System and Docker configuration, this IP address varies.

edit host.docker.internal is recommended from 18.03 onward.

Here is an example of communicating with the Docker host from a container on Mac:

# Dockerfile

FROM python:3.7-alpine

RUN ping host.docker.internal -c 4

sending build context to Docker daemon  562.7kB
Step 1/2 : FROM python:3.7-alpine
 ---> a93594ce93e7
Step 2/2 : RUN ping host.docker.internal -c 4
 ---> Running in 54ce465f7386
PING host.docker.internal (192.168.65.2): 56 data bytes
64 bytes from 192.168.65.2: seq=0 ttl=37 time=0.735 ms
64 bytes from 192.168.65.2: seq=1 ttl=37 time=1.898 ms
64 bytes from 192.168.65.2: seq=2 ttl=37 time=0.720 ms
64 bytes from 192.168.65.2: seq=3 ttl=37 time=0.643 ms

--- host.docker.internal ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.643/0.999/1.898 ms
Removing intermediate container 54ce465f7386
 ---> b2f32c26a135
Successfully built b2f32c26a135


How do I return just the full path of a file found with the find module in Ansible?

Answers to some parts of your question are:

"I want to find a softlink (to another file) in the $PATH environment variable on Linux managed nodes"

Use "shell" module instead of lookup.

  Lookup occur on the local computer, not on the remote computer.

"I want to extract the path of the file name for the file that is found. How do I do this?"

Use "files_found.files" to see the complete path. Checkout the return values section in find module documentation.
As far as i understood, you want to do something like this

Try out the following:

- name: finding files matching "goodfile" pattern
  find:
     paths: "{{ some_path }}"
     patterns: "goodfile"
     file_type: link
  register: files_found

- name : get the complete path
  set_fact:
    files_found_path: "{{ files_found.files }}"

 - debug:
     var: files_found_path


How to communicate queue-based processing delays to non-technical team members?

As with any outage communications, a non-technical reader will be primarily looking to understand:

How long was it?
How bad was it?

Amazon CloudWatch metrics provides the following metrics for SQS queues that can help answer these questions:

NumberOfMessagesSent: The number of messages added to a queue.
NumberOfMessagesReceived: The number of messages returned by calls to the ReceiveMessage API action.
ApproximateNumberOfMessagesVisible: The number of messages available for retrieval from the queue.

When graphed correctly, these metrics can be powerful visual aides in describing queue processing delays. Here are a couple of examples from an outage I experienced where the job's capacity to process queue messages was severely degraded:

NumberOfMessagesSent & NumberOfMessagesReceived

Graph Type: Line Graph
Statistic: Sum
Period: 5-minutes

This graphs the contrast between messages sent and received, which helps isolate which processing component is responsible for the delay. In this graph, the received metric drops dramatically while the sent metric continues on its normal trend, so we can infer that the issue lies in the queue reading component instead of the queue writing component.

Does this answer how long/how bad event was? Yes; describes processes impacted over time.

NumberOfMessagesReceived & ApproximateNumberOfMessagesVisible

Graph Type: Stacked Area Graph
Statistic: Sum
Period: 5-minutes

This graphs the queue depth on top of messages received, which helps show how far the queue backed up and how it recovered. In this graph, we can see that the queue depth backed up dramatically while the queue reading component was having issues, and began recovering when the queue reading component began reading messages again.

Does this answer how long/how bad event was? Yes; describes messages impacted over time.

Graphing Discussion

In both graphs, queue processing can generally be considered healthy when the lines overlap, and unhealthy when the lines diverge.  This is an easy pattern to teach to a non-technical team member, and can help them to quickly dissemenate where and how there are problems when presented with these graphs.

To further communicate specific points in graphs, you can simply annotate them:

Graphing Tips:

Label units and axes.
Use consistent colors for matching metrics across graphs.  Note that NumberOfMessagesReceived is orange in both graphs; this will help visualize the same metric across different graphs.
Vertically align graphs that describe similar metrics so that they are easier to compare across time.

Note: I've formatted these graphs for presentation on StackExchange, so these aren't necessarily how I would present them in an outage post-mortem. I have explicitly removed values from the left axis here to obscure them from StackExchange; you'll want to keep them in your post-mortems.

Additional tips

Empower Your Team: After training your team members to read these graphs, there is no reason keep them hidden away. Consider setting up a CloudWatch Dashboard and giving your non-technical team members read-only IAM access to CloudWatch, so that they can view these graphs any time.
Set Up Notifications: Consider setting up Cloudwatch Alarms based on the ApproximateNumberOfMessagesVisible metric if it exceeds some agreed-upon high value, and subscribe team members to notify them of potential issues. Cloudwatch Alarms have description fields that are sent along with the notification emails -- make sure to include a human-readable description to help your non-technical members disseminate the alarm.
Explore Other Data: Per Evgeny's comment, explore other data beyond what CloudWatch provides and think about how you can convey that data to your team.  His example of using message lifetime in the queue to create a histogram is a great example of this creative thinking, and can be accomplished by logging both the message send and message receive times in your application.  You can get the message Sent Timestamp via the SentTimeStamp Attribute on each queue message of the ReceiveMessage API response.  More details here.


Jenkins returns 502 gateway error and (104: Connection reset by peer)

If the system recently rebooted and your proxy server is giving 502/503s, it's most likely that the backend service failed to start.

Using whatever tools are most appropriate for your OS/distro (e.g. systemctl, service/chkconfig, stop/start/restart/etc.), do the following things:

Verify that the backend (in this case Jenkins) is configured to start on boot.  Enable it if not.
Start the backend.
If the backend fails to start, check the application logs to find the source of the error.  For Jenkins on Linux, this would be /var/log/jenkins/jenkins.log.


Instead of Continuous Integration and Continuous Delivery, I would define the definition of done as "Active Continuous Improvement at all levels of the organisation".

The other topics like automated builds, CI/CD, etc... these are just milestones and most definitely not the ultimate goal for the organisation in its DevOps efforts.

It does look like it might be "finished all DevOps" once you have an automated system with continuous integration. But in fact, even should you reach this stage, you will find plenty more to do.

Getting to a state where most people in the organization are constantly looking to continuously improve is extremely hard. Inertia often gets in the way, both existing inertia, and new inertia once some improvements were accomplished.

There is a lot to learn from Lean and from Theory of Constraints in this aspect. Both bodies of knowledge focus on the process of continuous/ongoing improvement and how to establish this process in an organization.


Kubernetes used to orchestrate Containerized Machines

You should look at DaemonSets. You can configure it so that the Pods runs on each node of your Kubernetes cluster. Each individual Pod (running your Linux machine) can be exposed to outside network using the IP address of the node on which it is running.

My need is more targeted to expose an entire machine on a precise static IP and not just a port.

Before getting into further explanation, I should point out that there is nothing like exposing entire machines.
Access to any machine is always through a combination of IP address and Port. So you should decide based on which port you would like to access on the container.
Example:
Let's say I have a 3 node Kubernetes cluster, I can create a DaemonSet with the application/image of my choice. Below names/details are imaginary, but should give an idea on the mapping.
ubuntu-daemonset-pod-xyz    k8s-node-1    192.168.2.1
ubuntu-daemonset-pod-abc    k8s-node-2    192.168.2.2
ubuntu-daemonset-pod-hij    k8s-node-3    192.168.2.3

The Pods above can be "exposed" on the IP address of the node on which it is running. That is: ubuntu-daemonset-pod-xyz on 192.168.2.1.


AWS Inspector "High Risk" : port open to the internet with IP whitelisting

If it is FTP (and not FTPS) then yes, it is a high severity risk. You are sending unencrypted traffic over the open internet. Even if you are whitelisting specific IP addresses, the traffic is still unencrypted between those specific instances.
In addition, FTP is a very old (created in 1971) and non-secure protocol. It is not meant to be a secure method of transferring files. I recommend you research some of the security concerns with FTP and switch to either FTPS or more preferably, use a more native AWS approach for transferring files between instances. If your organization is under any regulation (FEDRAMP, HIPPA, etc.), using FTP across the open internet will not meet the compliance requirements.