I Googled "Push on Green" and the first link was:
https://www.usenix.org/conference/lisa14/conference-program/presentation/klein

This was representative of almost the entire first page.  It looks like this term originated in Google's SRE group and has been taken up by the industry at large.

You are correct- "push on green" means that deployments are automatically executed when all tests pass.  This could be considered synonymous with "Continuous Deployment."  The accepted method for automating deployments to environments is to automate testing.

If your automated testing is rigorous enough, you can deploy any code for which the tests pass.  In very mature enterprises (the DevOps unicorns of all shapes and sizes) this can lead to Continuous Deployment all the way to the production environment.

My personal experience is that reaching this level of automated testing is a challenge in any enterprise.  I am currently in the process of pursuing this level of maturity in my current position.  Our first major milestone will be "push on green" deployments to any environment beyond "Integrated Development" (our "lowest" environment.)

This challenge is both technical and cultural.  As an IT organization, we owe it to our business stakeholders to prove that our testing is in fact rigorous enough to serve as the sole gateway to a deployment.  Once we have done that to our own satisfaction, we have to convince those stakeholders to let us try Continuous Deployment to the next environment (for us, this is QA.)


Tool preferences for rapidly spinning up and spinning down k8s clusters

You could try this:

https://github.com/heptio/aws-quickstart

Their CTO uses it to spin up K8S clusters for his weekly youtube live stream called TGIK, in order to showcase whatever he wants to talk about.

I myself use GKE so I do not need it.


Terraform - how to modify a existing S3 bucket that was created from module?

You'll have to modify the underlying module to make changes here.

Be sure you're aware of the impact this will have on other modules using your s3_website module. You might want to start using module source versioning.


How to get code from Bitbucket via Octopus?

It appears that Octopus supports a push from BitBucket, but not a pull (https://octopus.com/blog/bitbucket-pipes-and-octopus-deploy)
Github supports either option: https://octopus.com/docs/packaging-applications/package-repositories/github-feeds


org.jenkinsci.lib.envinject.EnvInjectException: java.io.IOException: remote file operation failed

The EnvInject plugin is basically attempting to load the build environment which uses configuration of JDK tools in Jenkins which fails during reinstallation of Java JDK library on the Windows slave (as per given configuration).

This JDK configuration can be found in Global Tool Configuration (at /configureTools) for the given JDK profile which was selected in that job configuration. The installation process failed to reinstall Java JDK due to running Java process in the background (such as java -jar slave.jar) which was using these DLL files.
So it sounds like it's not possible to reinstall the same Java JDK files which are used to run the Jenkins slave process.

The solution was to disable Install automatically option in Global Tool Configuration, as Java JDK was already installed on that slave.

Related: JENKINS-46989 - EnvInjectException: remote file operation failed.


I'm using the following ruby script to encrypt/decrypt chef data bags locally:
https://gist.github.com/anonymous/4923ebeee53fe4d43a1cfb70a2abaadd

to encrypt:

ruby encrypt.rb PATH_TO_SECRET_FILE < UNECRYPTED_DATA_BAG.json > ENCRYPTED_DATA_BAG.json

and to decrypt:

ruby decrypt.rb PATH_TO_SECRET_FILE < ENCRYPTED_DATA_BAG.json > DECRYPTED_DATA_BAG.json


jenkins build overview page not showing stages in webui


ansible-playbook: how to dynamically, conditionally set a large deployment environment (variable space) in a playbook?

Yes, an input host file for ansible looks like https://pastebin.com/Vtfw5FtV
The upper part node stuff is generated using terraform, we can dynamically up/down scale the number of nodes with terraform and terraform outputs the yaml used for the hosts part of the file. The tenant-specifiek yml vars (about 150 additional lines of tenant specific config yml) are kept in a separate git location and a jenkins step/python concatenates them together and pushes them to the host directory location for each tenant and runs individual roles (hotfix) or all of 'em (full deploy) or perhaps just stop/start. The Jenkins job is pretty much a control screen with a lot of dropdowns, radio buttons and selectors. We can select how many nodes we want for a single service, which build to grab from nexus, do a full deploy, a single service hotfix or just a configuration hotfix. All this is selectable/executable per environment, per tenant and per individual module/service. The Jenkins screen is dynamically build from the host (targets) & roles (actions) directory.
Again, we're most likely abusing Ansible "best practices" a fair bit but for us this approach has been working remarkably well for last >3 years in terms of overall manageability and gives us maximum flexibility of the whole CI/CD pipeline.


What Ansible does in cases there's a conflict between OS version and some software Ansible manages?

Ansible is at its core a programming language - a meta programming language written in python compiling itself into python and shell scripts mostly. It will do whatever you program it to do. It all depends on how your roles and playbooks are written. It certainly can be done - almost anything can be done - your install roles could track dependencies on package versions or ranges, obviously those might need to be updated over the years, but you can be safe if you put the work into it.

It is probably unreasonable to plan further than the End of Life of the LTS distribution though, as you will fail to receive new security updates and that instantly invalidates the system as production ready.


DevOps and agile aren't same thing but they are complimentary to each other. Agile development is development that adheres to the principles stated in The Agile Manifesto. In brief, agile is the word for an environment in which the priorities, according to the authors of the manifesto, are-

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation and
Responding to change over following a plan.

The point of DevOps is to deliver technology to business units in a timely fashion and ensure the technology runs without interruption or disruption. To achieve it's goal DevOps mostly focus on-

deep communication between software development and IT operational groups
automated deployment processes

Below are list of relation/differences with agile and DevOps-

Agile software development is a methodology for developing software, but DevOps, on the other hand, is all about taking software which is ready for release and deploying it in the safest, most reliable manner possible.
Agile is all about development. Sometimes, it takes over the entire company. Even when it does, agile discipline doesn't inevitably lead to DevOps. The practice of DevOps involves a separate discipline and methodology from those of agile.
Agile practices, like Continuous Delivery practices, can be part of a DevOps activities.


CI pipelines, concurrency and build results possibly overwriting each-other

I am using AzureDevOps and the trick we use when naming containers is to add build number to the label

namespace/application:branch-name-bulidNumber

So in your case, you could try gitlab variable CI_CONCURRENT_PROJECT_ID

After comment

for images clean-up you can use:

  docker image prune --all --filter until=48h

As docker system prune and docker image prune have the until filter. So docker image prune --all --filter until=48h would remove all (not just dangling) images that were created more than 48 hours ago. Hopefully that helps. source here


From what I can see you have done everything correctly, you have stored both the certificate material (i.e. the PFX) and the password for the PFX in a secure location that presumably no one else can access.  Assuming you don't give anyone else permissions on your Azure tenant and you don't blindly accept pull requests that change azure-pipelines.yml then you are in a good place.

If you are feeling paranoid, then you could create a second pipeline using a private repository that is triggered when the previous pipeline completes successfully.  However, IMHO, I would carefully consider the risk you are exposing yourself to against the complexity and fragility of trying to tie two pipelines together.


What minimal IAM permissions are needed by AWS Chatbot so that it can show logs?

AWS Chatbot uses CloudWatch Logs Insights, so the IAM policy for Chatbot must include these two statements:
{
    "Sid": "QueryLogs",
    "Effect": "Allow",
    "Action": [
        "logs:StopQuery",
        "logs:StartQuery",
        "logs:DescribeQueries"
    ],
    "Resource": "arn:aws:logs:::log-group:/aws/lambda/*"
},
{
    "Sid": "GetQueryResults",
    "Effect": "Allow",
    "Action": "logs:GetQueryResults",
    "Resource": "*"
}

With these statements in place, pressing the logs button will yield an initial message from Chatbot like this:

I ran this CloudWatch Logs Insights query to fetch logs from 2020-11-18T09:02 UTC to 2020-11-18T09:07 UTC.

... as well as several follow-up messages containing a plaintext dump of the logs.


Automated tag versioning on unchanged docker images

How it should look like:

Commit changes in some microservice and push it to VCS origin repo
CI system sees new commit
CI starts build and tests
If everything's good, it pushes image to docker registry with tags

So, the answer is: CI should know, what service has changes.


per-merge request GitLab CI pipeline customization

I'd go with what gitlab already offers, the only/except keyword.

Something like:

test:e2e:A:
  script:
    - whatever you do to test A part
  only:
   refs:
      - master
      - /^A-.*/

test:e2e:B:
  script:
    - whatever you do to test B part
  only:
   refs:
      - master
      - /^B-.*/

The only requirement is that work on A or B is done in branches prefixed with A- or B-.

the refs keyword accept regexes to match, more details on gitlab-ci docs here

If your gitlab is pretty recent (>12.3) the rules method would be prefered and the example above for A would become:

test:e2e:A:
  script:
    - whatever you do to test A part
  rules:
    - if: $CI_COMMIT_REF_NAME =~ /^A-.*$/
    when: always
    - if: $CI_COMMIT_REF_NAME =~ /^master$/
    when: always

Or with a single regex:

test:e2e:A:
  script:
    - whatever you do to test A part
  rules:
    - if: $CI_COMMIT_REF_NAME =~ /^(A-.*|master)$/
    when: always

I didn't test those entries above, let me know if one fail


Can I run an API gateway on the same instance as my API?

It should be fine to run the API Gateway on the same server as API-apps - for smaller setups. For larger setups when you need more servers may make sense to use dedicated nodes for API Gateways. API Gateways are increasingly playing a dual role as an application load balancer, or Level 7 Proxy for your APIs - those have a specific role when it comes to network addresses and routing, this is the main reason to why it makes sense to use dedicated servers for this part of your stack.

why an API gateway should be on a separate server to the API?

what is more important for this part of the stack is that you would like to avoid single-point-of-failure, so you should use more than one instance for API Gateways, and also for the API apps. When using multiple instances, it is often easier to separate the load balancer/Gateway from your app workload due to the specific needs for network addresses for the gateway, but in the end this depends on how your whole architecture looks.


AWS subnets for lambda synchronous invocations

I don't entirely understand your use-case, but it sounds like you are trying to limit the number of Lambda invocations by restricting its IP range.
There are two better ways to rate limit your application. Option one would be to use API Gateway request throttling:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html
Option two, Lambdas have a concurrency limit that can be set to achieve a similar behaviour to your approach:
https://aws.amazon.com/about-aws/whats-new/2017/11/set-concurrency-limits-on-individual-aws-lambda-functions
Hope that helps with your design!


I don't think you can alter the trigger right now in Azure DevOps. What I do is to create an extra task to start with, and use powershell (or your language of choice) to check if the week number is even or odd and then proceed with the build or not.


Is is possible to configure settings in Bitbucket globally?

At the moment this seems not to be possible in bitbcuket at the moment. Therefore I have created an enhancement in the Bitbucket issue tracker.


with_fileglog creates a loop with the fileglob lookup. As all other lookups it runs on the controller machine, not on the remote server.

Since I guess you don't already have those backup files on the controller in the path you referenced, your loop is empty.

As you have guessed, this approach will not work for your scenario. I suggest you have a look at the find module to create the list of files to fetch from your primary server.


Feature flags are an engineering device that can be used to avoid long-lived branch and conflicts in product development.  Here is how it can be used the context of an object-oriented language to help developers collaborate on a specific product feature while one handle a new version. This solution can also be used in non object-oriented contexts, provided a notion of “interface” exists. (cf. OCaml module system.)

For the purpose of illustration, we assume a tool presenting reports about data stored in a database. The code implements a DatabaseClient class used to perform requests. As the dataset grows, it becomes clear that some alternative data layout would improve the application performance. Therefore Alice will develop a new version of the DatabaseClient able to retrieve data from the structures with improved layout, while Bob will maintain the historical DatabaseClient.

With the following steps, Alice and Bob can collaborate on short-lived branches while minimising their conflicts.

Alice rename DatabaseClient to DatabaseClient_v1 and create a delegate class called DatabaseClient that uses an object DatabaseClient_v1 and implements an interface called DatabaseClientInterface. (If possible, this DatabaseClientInterface should be a code artefact but duck-typed languages not always support this.)
Bob reviews changes made by Alice in 1 and is aware that his maintenance job should happen on DatabaseClient_v1.
Alice introduces a new configuration flag in the application that governs the behaviour of the DatabaseClient delegate and implements a DatabaseClient_v2 placeholder, a class implementing the DatabaseClientInterface whose methods all throw a “Not implemented” exception.

After this, Alice and Bob can collaborate without explicit synchronisation, because code written in their respective iterations is subject to the DatabaseClientInterface. This minimises the risk of a conflict resulting from their concurrent work.

Iterations from Alice can be very short, like implementing a test, implementing a method, or even partially doing so, because in production, the code is not selected for use and does not need to be fully functional. The automated testsuite should be configured so that the DatabaseClientInterface always uses DatabaseClient_v1 while Alice can easily toggle to DatabaseClient_v2 when running the testsuite locally – or in a custom CI setup. Once everything is ready, a single commit can perform the change, by updating the configuration value governing the  DatabaseClient delegate.


What percentage of microservice architectures employ polyglot programming models?

A good source for statistics could be stackshare.io.

In general, polyglot services i.e. teams delivering different stacks could create silos, and additional overhead for diverse toolchains; nevertheless I consider that larger organizations have the polyglot situation organically because this type of thing is not always an issue of top-management.

Arguments against polyglot approach:
"Why You Should Think Twice Before Even Considering Polyglot Microservices"

Worth to mention is the polyglot toolkit Eclipse vert.x (though I am not sure how is this different to say "it has bindings to many languages").


Why do some components vanish from the catalog list if one component has been installed in Rancher?

You haven't specified anything concrete, so I can't give you a specific reason...  But catalog items can:

Be mutually exclusive (e.g. you can't use ipsec and vxlan at the same time)
Be deployed at most once in an environment
Require the environment be a specific orchestration type
Require a certain range of Rancher versions
etc


I don't view DevOps as a job position. Yes, I also have job offers out there for titles like "DevOps Engineer" [sic], but that just means I am looking for people who mainly can take a team with little DevOps knowhow and guide them along (be it through just doing the bulk of the work, or evangelism or whatever). But at the end of the day, everybody on the team needs to have a working understanding at least of what's going on.

DevOps is more of a feeling, a culture, and a set of common tools and techniques. A guideline if you wish. With memes like having the same environment across dev, qs, prod, immutable containers being shipped across all those, "everything as code", applying Agile and Lean to ops, having devs and ops in the same team and such.


Deploying a single-page web application with a java backend on kubernetes and haproxy

Modern way to achieve seamless roll-out in this scenario is not on the load balancing (i.e. haproxy) side, but instead via proper deployment planning, backward compatibility and/or feature flags.
The issue with load-balancer tinkering is that in the end of day even if you get sticky sessions right it's not going to solve your problem. The reason for that - imagine a user that opened your app in a browser with old react. Underneath, you deploy new java back-end. If the user doesn't reload her browser - she's still going to use old react app with new back-end no matter what (just because javascript is already loaded to the browser).
Instead, look at following example how you can achieve it using API versioning:

Let's assume that data model on one of your api end-points is going to change
Let's say your old api was /v1 (or even un-versioned)

Here is how you can solve it:

You create /v2 version of your api, register /v2 version in your react app
Your latest java back-end needs to still have both /v1 and /v2
You first deploy all java back-ends to your back-end deployment with /v2 support (all your UI is still /v1 - but this is fine, because latest back-end supports it)
Once all back-ends are deployed, you can start roll-out of your UI deployment with latest image (again, experience is seamless since all back-ends support both /v1 and /v2)

As you can see, in the above approach, version conflict is avoided and experience is seamless.
P.s. You might want to consider splitting your docker image in 2 - it's a little odd to have a multi-purpose image with 2 entrypoints. There may be little justification for this design.


It is the address of the DNS server (probably CoreDNS but possibly kube-dns) for the cluster.

Kubernetes uses a DNS server within the cluster so pods can find each other using service names. This is the "cluster DNS" server. Every time a service is created, it gets registered in the DNS server.

In Linux, the /etc/resolv.conf file is where the DNS server is configured. If you want to use Google's public DNS server (8.8.8.8) your /etc/resolv.conf file has the entry nameserver 8.8.8.8.

In Kubernetes, the kubelet process on a worker node configures each pod.  Part of the configuration process is to create the file /etc/resolv.conf and specify the correct DNS server.  That server is spec'ed by the clusterDNS configuration option.

How you set it (if you even need to set it yourself) depends on how you're installing kubernetes.  It ultimately depends on the CNI provider you're using, but also if you're doing "kubernetes the hard way" versus kubeadm.


Entering to docker container of my elasticsearch google kubernetes pod - CONTAINER ID is changing

If I understand you question clearly, you're trying to mount /jaeger-data inside the elastic-search container or pod. To achieve this, what you can do is put this mount details inside the statefulset yaml of the Kubernetes pod for elastic-search. Using the volumeMounts inside the statefulset, you will not need to ssh or exec inside the container or bother about the change or container ID for each pod restart/recreation. You can use the below example as a reference for your PVC and volumeMounts, just edit your statefulset using kubectl edit sts elasticsearch-0 -n default, this will open an editor and you can replace you PVC details in the below yaml and append and save and exit will recreate your pod and attach the mountpoint:

          volumeMounts:
            - mountPath: /es-storage
              name: jaeger-data
  volumeClaimTemplates:
  - metadata:
      name: es-storage
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi

Container ID will change with each recreation of containers;
If in case you need to exec into the container, the better way is to use kubectl exec -ti <pod-name> -n <namespace> bash to exec into the container from any of the master node(similar command as docker exec, just replace pod-name and namespace) or from your local laptop if you have kubeconfig configured.
Using Service, you can expose this application on a particular port, Kubernetes creates an abstraction and itself takes care of the change of pod IP


What is the duration for the App Engine task queue tombstones?

TombstonedTaskError is raised if you're attempting to enqueue a named task using a name that has been used recently, see What is TombstonedTaskError from App Engine's Task Queue?.

The duration of the de-duplication logic has been documented since that answer was posted. From Naming a task (emphasis mine):

  When you create a new task, App Engine assigns the task a unique name
  by default. However, you can assign your own name to a task by using
  the name parameter. An advantage of assigning your own task names is
  that named tasks are de-duplicated, which means you can use task names
  to guarantee that a task is only added once. De-duplication
  continues for 9 days after the task is completed or deleted.


Since you've tagged this question as "culture", I'm assuming you're not interested in a specific application, but in the broader questions of workflow and management.

I'd probably start with "The DevOps Handbook"; it's a good overview of different things to consider, without diving too deep.

"Continuous Delivery" by Jez Humble is also often referenced; I haven't read much of it yet, but it covers the concepts of source control & automation of builds.

If you're starting to get into applications at scale (this may be too much of an assumption), another good book is "The Practice of Cloud System Administration" by Limoncelli et al.


Ansible - Variable expansion inside hostvars declaration?

No, you cannot nest jinja2 expansion i.e. the following won't work:
# Warning ! Those will fail !
some_var: "{{ some_dict[{{ some_dynamic_key }}] }}"
other_var: "{{ {{ dynamicaly_prefixed }}_var }}"
# Did I forget to say the above examples will fail ?

But you can use your var without any problem, just as in almost any other programming language dealing with list and indexes. There is actually an ansible FAQ entry on this particular subject
some_inventory_hostname: 192.168.0.162
node1_hostname: "{{ hostvars[some_inventory_hostname].node1_hostname }}"


kubernetes: how to set "top node" and "top pod" permissions?

Thanks to arghya-sadhu's comments on stackoverflow, I've got it working:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: get-pod-and-node
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "watch", "list"]

(I'm aggregating to the standard 'view' role)


How to mange data access in a CICD pipeline for multiple users with different permissions

My general experience is that CI/CD tools don't support impersonation or claims based authentication well or at-all. In addition you probably want the CI/CD tool to have more access than your users, because you want them to only use the CI/CD tool to change things.

The way I have handled this in the past, however, was as follows:

Have the pipeline call the API of the CI/CD to find out who ran the pipeline
Use this pipeline to look up the group's that the user is a member of this (i.e. in LDAP)
If the member is of the appropriate group, continue. If not abort the build.

It's quite important to abort rather than fail the build because then it differentiates between an invalid user running the build, and a valid using running a build which then fails.

Another approach would be to sync LDAP groups with your CI/CD platform, and set permissions that way, although that may not give you the granularity you want.


Unable to connect to remote service using AWS VPC Peering

How are you trying to connect to the remote service? - IP address, default internal/external DNS name or custom DNS name private/public hosted zone?
Please compare the output of nslookup to the remote service in both the VPCs. If the IPs resolving are different, the traffic is trying to go through the public internet and not the peering connection.
It would most probably be the need to either share the private hosted zone with the peered VPC or share internal hostnames across the peering connection


How do I make a redis container accessible through bridge networking from the host?

By default Docker uses the bridge network.
On a default Docker installation you can simply map a port to the container's service port.  The Redis Docker image exposes its service on port 6379 so you can do:
$ docker run --name rd -d -p 6379:6379 redis:latest
From your host you can now run use the redis-cli utility:
$ redis-cli
127.0.0.1:6379> ping
PONG
127.0.0.1:6379>


I resolved this issue by removing the vendor folder in the Laravel package.

Once the vendor folder is removed, the composer.json file is automatically used to run the installation. All necessary files are then downloaded and installed.

This removes the need to run the 'composer install' command post installation.


How do I work around the fact that AWS SQS is not HIPAA compliant?

According to Amazon AWS

  Customers may use any AWS service in an account designated as a HIPAA
  account, but they should only process, store and transmit PHI in the
  HIPAA-eligible services defined in the BAA. There are ten
  HIPAA-eligible services today, including AWS Snowball, Amazon
  DynamoDB, Amazon EBS, Amazon EC2, Amazon Elastic MapReduce (EMR),
  Amazon Elastic Load Balancing (ELB), Amazon Glacier, Amazon Relational
  Database Service (RDS) [MySQL, Oracle, and PostgreSQL engines only],
  Amazon Aurora [MySQL-compatible edition only], Amazon Redshift, and
  Amazon S3.

source: https://aws.amazon.com/compliance/hipaa-compliance/

This means that as long as you are not storing or transmitting PHI in SQS, just the information about where this PHI is being stored - you probably can pass an audit reg. HIPAA compliance.

In the architecture you describe, the SQS queue does not need to include any PHI content. This would make it comply with the above statement.

More information about HIPAA compliance on AWS is available in this whitepaper from January 2017 - https://d0.awsstatic.com/whitepapers/compliance/AWS_HIPAA_Compliance_Whitepaper.pdf

Specifically SQS is mentioned and explained in the HIPAA FAQ - https://aws.amazon.com/blogs/security/frequently-asked-questions-about-hipaa-compliance-in-the-aws-cloud-part-two/.

update: As of May 1st 2017, SQS is now HIPAA compliant. https://aws.amazon.com/about-aws/whats-new/2017/05/amazon-simple-queue-service-sqs-is-now-a-hipaa-eligible-service/


Wikipedia has a very good answer to this question. Artifact, sometimes also called Derived Object, is a product of some process applied to the Code Repository. Originally they were called Build Artifacts, but as more processes were applied other than build to create them, the first word was simply dropped.

The major distinction is that artifacts can be recreated from the code repository using the same process, providing you have preserved the environment in which the process was applied. As this process can be time consuming and the environment can be preserved imperfectly to be able to recreate the artifacts in the exact same way, we started to store them in Artifact Repositories.

Storing them apart from Code Repository in an Artifact Repository is a design decision a DevOps engineer would make. Some companies, namely Perforce, suggest to use their Code Repository as Artifact Repository as well. There are different requirements in terms of access, auditing, object sizes, object tagging and scalability on each repository and so depending on situation it is often better to use two distinct products. For example Git repositories are copied in their entirety to every development machine and so storing artifacts in the code repository would increase its size beyond all reason, although lately there are ways to mitigate this. Another decision to make is which artifacts to store. Some companies store even intermediate artifacts as individual object files, to speed re-builds, others store simply just the final binaries. Not all artifacts have the same value. Artifacts resulting from a release build could have different requirements than artifacts resulting from a developer build.

Most common artifacts are results of the following processes: Configuration, Preprocessing, Compilation, Linking, Automated Testing, Archiving, Packaging, Media files creation and processing, Data File Generation, Documentation Parsing, Code analyzing, QA, etc.


Jenkinsfile credential as paramter unable to use it

withCredentials publish environment variables, and in sh to access the environment variables, you have to do it like this: "${env.JOB_BASE_NAME}".
Try this:
withCredentials([usernamePassword(credentialsId: params.git_credential, passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
   sh 'git reset --hard; git clone https://${env.DOCKER_USERNAME}:${env.DOCKER_PASSWORD}@repo_url/scm/${params.repository}.git --branch master'
}


Error starting cloned VM in Vsphere Client

Have a look at this page which documents EnvironmentBrowser for clues of which setting to edit.

Possibly something has a relative path that is now modified, or a resource that's fully used up or no longer available (or no longer valid, but that makes little sense if it's the same VM host).


How to model Man, Machine, Measures, and Process in a DevOps world?

What they are talking about is Kaizen 5M (Man, Machine, Material, Method, Measurement). It is approach to continuous improvement at every station in the process and the Ms are possible points of improvement and to which there is a corresponding question (5Qs). Sometimes Environment is added for 6th, like in this process that explains how to construct the questions using the Ishikawa diagram. These are pretty much essentials of the TPS / Lean Manufacturing. But the improvements are not in utilization, they are improvements in quality. You never strive for utilization as that is counterproductive to the throughput of the system.

It is important to understand that Man, Machine, Material, Method and Measurement are not easily separated. Sometimes Machine, Material and Measurement come on one side together and Man and Method on the other side. As you can replace a Man and a Method on that work station.

In terms of software development, you have Software (Machine), Issues (Material), Code Quality/Acceptance (Measurement), Man (Programmer) and Method (Development Process). The man trains on the machine and becomes familiar with it, with the material being worked on, understands the measurement required, learns the process. All those are living in the Man's brain and so are not easily separated once learned. Changing a Method is only possible if the Man has not fully internalized it yet, otherwise it gets easier to change Man and Method. Also Machine, Material and Measurement are often tied together through automation and configuration. That is why those are being drawn on opposite sides of the diagram.

If you read the book carefully, it does not really talk about utilization other than on the bottleneck of a value chain. In order to elevate and exploit the bottleneck. Several methods are employed for that in the book, including Kanban.

You don't want to optimize the individual stations of your process (Customer->Support->Development->Review->Testing->User Acceptance->Deployment->Customer), but you need to model the transitions between those work stations, their dependencies and to monitor Work In Process (WIP) moving through the system. Usually through an Issue Tracking Software (or Kanban system), which is equivalent to materials tracking in manufacturing. Where the WIP piles up in front of the work station in your critical chain process, you will find your bottleneck and that is the place where you want to optimize using Kaizan (5Ms,5Qs)

Notice: I've added Customer both at the start and the end of your process, as each value chain has to start and end with a Customer otherwise it does not represent value to the company.


How to install Certbot via Ansible playbook?

Here is the direct Ansible translation of your bash install script:

- apt_repository:
    repo: 'ppa:certbot/certbot'

- apt:
    name: "{{ item }}"
    update_cache: yes
  with_items:
    - nginx
    - python-certbox-nginx


Advantages of using Code-pipeline and Codebuild for CI/CD over Bitbucket pipeline?

A bit too long for a comment, so converting to an answer:
That's super broad and highly depends on your final needs IMHO, if you're full AWS and ok with being vendor locked down, then code commit/code build/code pipeline is great as it's nicely integrated now.
If you have needs outside pure AWS services, that may get harder, for exemple if you plan on stretching on multiple cloud vendors in a always on DRP (Disaster Recovery Plan), that's probably not the better tool.
My take is that for a full AWS things, AWS products are good and usually soften the edges, but if you need to interact with things out of AWS, there's solution more aimed at CI/CD like Bitbucket, as you said but that'd be my last choice, CircleCi, Travis, Github Actions and Gitlab.
The CI/CD cost is rarely a concern, even if you have a super large company with dozens of products, the code repository and user management costs will probably out-stand the pipeline costs by a large factor unless you run thousands of pipelines per day.
My personal preference goes to Gitlab for its versatility in pipelines, how easy internal runners are to setup and to avoid vendor lock down on the hosting.


How to use Jenkins Pipeline and choose triggers for email notifications?

This seems not to be possible at the moment with pipelines. See this Jira ticket.


Docker: PHP deployment in apache, couldn't acces container URL in local browser; shows blank page

You need to expose the container ports locally using either:

docker run --rm -it -p 8080:80 myapp
or a docker-compose.yml file:

services:
  myapp:
    image: myapp:tag
    ports:
    - 8080:80

Then, you'll be able to curl localhost:8080.


What is a good strategy to keep my site online when S3 goes offline?

In March 2015, Amazon AWS announced they support S3 replication across regions. When a certain region in S3 goes offline, you can serve files from your mirror in another region.

source: https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/

The practice of keeping your infrastructure online by doing a switch over to another region is a complex one, but S3 is a relatively small and simple component. Netflix has a great article on their experience with Chaos Gorilla.

This also applies to service degradation, like increased latency. Not just when a service you depend upon is completely offline. Netflix has an article on this as well: Chaos Engineering Upgraded.


That's a known issue .. since image layers are built using containers, data saved to folders does not make it to the next layer or your final container

Removing intermediate container b3ff83bbf273

If you want data to persist between updated containers you have to manually map data to the host outside the container-specific mount points.

For example, when creating a container use the -v option to map a volume:

docker run -d -v /var/container_data/mysql:/var/lib/mysql mysql_image

For a detailed documentation
https://docs.docker.com/engine/reference/run/#volume-shared-filesystems


I think it might be a good idea to update deployment.yaml for each version/release and commit to git so that later, any other person may follow through and even revert back to an earlier release/version.


How to integrate docker-compose or docker run command arguments into SCM and Jenkins build?

If you have docker compose installed on the Jenkins Build machine you can use sh docker-compose up to run docker-compose from a new shell.

In general, you can run any command from your JenkinsFile using the sh command.


Yes: in the hello world CDK example, they call it "synthesizing":
cdk synth
^^ produces CloudFormation file.


Are there cheaper alternatives to cloud watch logs for creating dash boards?

Grafana was not really built to view logs from the get go.

They however introduced this feature in v6.0 via the Explore tab

This can be leveraged in 2 ways.

Via Elasticsearch data source (in v6.3+)

Check the Elasticsearch docs for more.

If you want, you can still integrate this with Cloudwatch. Check out the log subscirber-consumer solution & this

Via Loki & AWS Lambda

If you want to stick to Cloudwatch as your primary source of logs, you can create a Lambda function that ships them to Loki which in tern can be used in Grafana as a data source. See 1, 2 & 3


What makes a difference between high-cost cloud providers and low-cost ones?

@jasper, and welcome to DevOps SE!

While your statement can be confirmed by other voices, it's very hard to x-ray price policies of a company. It's a complex gameplay always driven by many factors and decisions and a question about pricing policies of Digital Ocean vs. AWS might be a good and on possibly well on topic question at Economics SE.

For example, one could argue to find facts supporting the evidence of Digital Ocean currently being cheaper that:

Docker, Inc. is in a situation where it cannot afford losing customers, so maybe they a cautious and prefer lower margin to attract customers.
Amazon is the market's prime with top notch customers from overall industry and government: stability, seriousity and credability are way more important than money, especially given that expenditures on information technology is only a (small) part of industrial budget planning. Also, Amazon has many more data centers and is as whole investing billions to AI technology ($0.8b in 2017, almost complete estimated market value of Docker Inc.), custom data center hardware, buying startups etc.

Please consider making your question more specific in the DevOps SE site context to get more answers.

This sort of decision is possibly best to elaborate together with the customer, collect their business requirements if price is not the only one requirement and assess risks.

Further reading/related business topic: due diligence.

Additional note: one might think, AWS would have way more customers than Digital Ocean but at least as of 2017 this was not the case.

Digital Ocean: 1 million users in 2017
AWS: more than 1 million users in 2017


Watch folder, when direct changes are made push to repo

First, I'd highly recommend not doing this - a better approach would be to have a single source of truth (that is your repo). Force developers to commit & push changes and let the pipeline do the rest. If it's not in repo, it didn't happen.
If you absolutely have to, then I'd go with "brute-force" way - have a pipeline that triggers periodically (every couple of hours or so) and does a diff of server files against repo files. If something changed, send a threat email to the developer who did this sync back to repo.


Using DevOps pipelines to create distribution bundles of applications

Add a config file with parameters used in install/run script. You can look at Harbor (Github, install docs) as an example of doing just that (and they have a configuration migrator).


How to notate ipv6 addresses in inventory file?

This blog post covers all of the methods:

excerpt - Host names and patterns in Ansible 2

  Inventory hostnames

  Ansible 2 requires inventory hostnames to be valid IPv4/IPv6 addresses or hostnames (i.e., x.example.com or x, but not x..example.com or x--). As an extension, it accepts Unicode word characters in hostname labels. Any mistakes result in specific parsing errors, not mysterious failures during execution.

  Inventory hostnames may also use alphabetic or numeric ranges to define more than one host. For example, foo[1:3] defines foo1 through foo3, while foo[x:z:2] expands to fox and foz. Addresses may use numeric ranges: 192.0.2.[3:42].

  IPv6 addresses

  A number of problems with the parsing of IPv6 addresses have also been fixed, and their behaviour has been made consistent across the inventory (.ini files) and in playbooks (e.g., in hosts: lines and with add_host).

  All of the recommended IPv6 address notations (from spelling out all 128 bits to the various compressed forms) are supported. Addresses with port numbers must be written as [addr]:port. One can also use hexadecimal ranges to define multiple hosts in inventory files, e.g. 9876::[a:f]:2.

  A couple of small but necessary bugfixes go hand-in-hand with the parsing changes, and fix problems with passing IPv6 addresses to ssh and to rsync. Taken together, these changes make it possible to use IPv6 in practice with Ansible.

Therefore patterns such as these IPv6's would be allowed:

fully specified 128 bit address
[addr]:port
hexadecimal ranges - 9876::[a:f]:2
various forms of compressing ranges - FF01::[1:6]

Simplified examples:

# IPv6 Address                                # Simplified Notation
-----------------------------------------     ----------------------------
- FF01:0000:0000:0000:0000:0000:0000:0001     - FF01::1
- 2031:0000:130F:0000:0000:09C0:876A:130B     - 2031:0:130F::9C0:876A:130B
- 0000:0000:0000:0000:0000:0000:0000:0001     - ::1
- FE80:0000:0000:5EFE:0192.0168.0001.0123     - FE80::5EFE:192.168.1.123
- FE80:0000:0000:0000:1585:4868:495F:D521     - FE80::1585:4868:495F:D521

References

NETWORK BASICS: IPV6 ADDRESS SIMPLIFIED NOTATION


Terraform 0.12 - For each subnet-id in array, get cidr_block

data "aws_subnet" "subnets" {
  for_each = toset(var.aws_subnet_ids)
  id       = each.value
}

locals {
  cidr_blocks = [
    for subnet in data.aws_subnet.subnets :
      subnet.cidr_block
    ]

  cidr_blocks_rendered = [
    for cidr_block in local.cidr_blocks :
     {
       from_port   = 6443,
       to_port     = 6443,
       protocol    = "tcp",
       cidr_blocks = cidr_block
     }
  ]
}

output "output1-1" {
  value = local.cidr_blocks_rendered
}

This is what ultimately I needed.


Is zipping a file supposed to happen in Jenkins or on Nexus?

If you are building these artifacts for a release/deploy, you will want them to be in as ready to release of a package as possible. If that is as a .zip, then yes you should create the .zip on the Jenkins server, then upload to Nexus. Your deploy system will then download, decompress, and manipulate the artifact as seen fit. In general, I would only use the node running Nexus to store artifacts and not execute any "build-like" processes. That is exactly what Jenkins was made for.


Serve an application built by the pipeline

Gitlab CI/CD offers Docker Engine. If you can create a docker image, you can serve the application the same way as you were doing with Jenkins. You can find more details on how to integrate both here.

Hope it helps.


could you please redirect me to some article, blog, white paper or suggest me a book where I could learn and get a feel of what fully implemented CI/CD looks like.

Sure, look how Google does it.

  I haven't seen a fully implemented DevOps lifecycle ever.

There seems to be a misunderstanding here. "DevOps" is a lot of things; but a "full" CI/CD (meaning each individual developer commit turning up in production fully automated shortly after the commit - continuous deployment (not only delivery)) is by far not the norm. In my experience with a few larger local and national customers, where production or business critical things are controlled by the software, and there usually are large amounts of critical data involved, it always stops before the production stage.

Full CI/CD is complicated. Yes, you can cobble the tooling together pretty quickly, but the prevention, detecting and backing out of unwanted commits is very hard. If you look at the above article about Google you will see that huge effort went into getting this to run (I seem to remember only their core DevOps team is about 100 persons, doing nothing than tooling; and then they invest a lot of work into peer reviews, knowledge transfer, etc. etc.). Whole ecosystems of components, procedures and mindsets were created specifically for this. The fact that we mere mortals get to play with things like Kubernetes is a result from that, their breadcrumbs if you will.

Even only the one fact that they are doing Trunk-based development would be too complicated (regarding processes, necessary quality requirements etc.) for many more normal, smaller teams.


Jenkins file that uses docker-compose gives ambigous error

You need to install the Docker Commons plugin? https://plugins.jenkins.io/docker-commons/

Edit: According to this link for I would like to use Docker CLI inside of a Pipeline job

  You need to make sure that “default” points to a docker tool inside of
  Manage Jenkins> Configure System which will also run on the os of the
  slave.


AWS ElastiCache Redis DNS error - Name or service not known

It could be due to a "VPC DNS throttling", you can find more information on the official documentation.
https://aws.amazon.com/premiumsupport/knowledge-center/vpc-find-cause-of-failed-dns-queries/


Cloudfront for ELB, automatic redirection

As the issue is happening only with /admin URL, this might be related with cookies which are being passed to the backend application's /admin URL.

You may need to check with CraftCMS team (or some other forums) and get the list of cookies which are being used by the /admin URL. You can whitelist those cookies in cloudfront specifically for /admin URL.


How to assign saltstack pillar values from Jenkins comma separated input parameters?

You didn't say how the sls file is executed.

One way would be to pass pillars on command line:

salt '*' state.apply ftpsync pillar='{"ftpusername": "test", "ftppassword": "0ydyfww3giq8"}'

Or you can set the group name as a grain on the minion in advance.
Yet another way would be to set the group name as a jinja variable in the sls formula, depending on the hostname grain. Assuming your host names are set and standardized.


How to pull a docker image from a private docker registry using Helm?

Based on this Github documentation it is possible to pull a docker image from a private docker registry:

values.yaml

imageCredentials:
  name: credentials-name
  registry: private-docker-registry
  username: user
  password: pass

templates/imagePullSecret.yaml

{{- define "imagePullSecret" }}
{{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry (printf "%s:%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }}
{{- end }}

templates/secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.imageCredentials.name }}
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: {{ template "imagePullSecret" . }}

at the end of the templates/deployment.yaml add the following:

imagePullSecrets:
  - name: {{ .Values.imageCredentials.name }}


docker container does not open port for php:latest

Try to use php -S 0.0.0.0:8080:

docker run --name php -v $(pwd):/app -w /app -p 8080:8080 -d --rm php:cli php -S 0.0.0.0:8080 -t . index.php

Explanation:

The localhost(or 127.0.0.1) address is attached to the container namespace, so anything outside of this namespace cannot have access to it. Furthermore, the 0.0.0.0 (in this context) is a specific address that means all IPv4 addresses on the local machine. If you check inside the container, you could see:

$ ip a
lo:
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
eth0:
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

The eth0 interface is the interface used when you forward the port. As you don't specify the IP address of the container, you can use the 0.0.0.0 to listen on the eth0 interface.

I don't know if I make my self clear, but you can (should) read this if you want to know more:

https://pythonspeed.com/articles/docker-connection-refused/
https://superuser.com/questions/949428/whats-the-difference-between-127-0-0-1-and-0-0-0-0


Is there a way to use a Jenkinsfile from a git-submodule in a multibranch pipeline?

I agree with Omri that you are heading the wrong way. I would recommend using a shared library which implements the pipeline. You would then have a simple Jenkins file which looks all the same. See https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/

Let's say you have a shared library which contains a vars/commonpipeline.groovy which contains something like this:

def call(body) {
    // evaluate the body block, and collect configuration into the object
    pipelineParams= [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = pipelineParams
    body()

    pipeline {
        agent {
            label pipelineParams.nodes
        }
        options {
            buildDiscarder(logRotator(artifactNumToKeepStr: '1', numToKeepStr: '10'))
            disableConcurrentBuilds()
            timestamps()
            timeout(time: pipelineParams.timeout, unit: 'MINUTES')
        }

        stages {
            stage('Build') {
                steps {
                    //TO SOMETHING
                }
            }
        }
    }
}

Then you can have a Jenkinsfile in your dev project which loads the library and uses the common pipeline like this:

import org.apache.commons.io.FileUtils
library identifier: 'shared-pipeline-library@master', retriever: modernSCM(
  [$class: 'GitSCMSource',
       remote: 'https://bitbucket/jenkins/hared-pipeline-library.git',
       credentialsId: 'bitbucket.service.user'
  ])

commonpipeline {
    nodes     = 'BUILD' /* label of jenkins nodes*/
    timeout   = '60'
}


Deploying a SaaS solution which is custom for every customer

I'd investigate what would it take to bundle all these customizations into something like a per-customer configurable theme that can be served by a single, more generic application code.

It probably won't be easy and it may take a long time to reach such capability, but it can be done gradually: start with the N*5 repos and, as you inch forward toward this ideal solution you'll see the deltas in the application code from the repos starting to shrink - the differences will gradually migrate towards these custom themes/artifacts.

When there are no more app code differences left then you can replace the N*5 repos with just 5 app code repos for now common app code and N*5 theme configurations/artifacts repos. Much easier to maintain.


How can Agile methodologies, CI/CD be applied to massive, monolithic software projects with 100s/1000s of developers?

Well, you won't be able to apply an Agile methodology to a large team. One of usual principle is to work on Pizza Team (Less than 10 persons which can share a large pizza for diner together) because the lack of formalism advocated by Agile process makes it hardly applicable to a large team, the information would be scattered and some would be lost in the crowd.

So before thinking about the CI system, you'll have to review the organization around the software you build.
This mainly need to refactor a large project in smaller independent pieces, this could be separate softwares (and you'll take the path of microservices) or it could just be an architecture design within the software, where a class/object/library is given to a small team. This team has to document its service input and output (service contract/service orientation).

Once this is done you can split the responsibility of each independent piece to a smaller team which will be able to apply an Agile process.
This may solve the CI/CD problem, splitting the code base would allow each team to follow it's own pace, as long as it's interface is properly versioned and that the input/output are fixed for a type of call you can make parts of the product evolve while some other part keep a previous behavior.

The main drawback is that the service contracts from each part must be compatible between versions in use, which need a dependency system to know which part use which version of what before dropping an unused version of an API. This brings problems to make breaking changes and usually need a careful transverse planning between producer and consumers.


Jenkins Plugin:Extended Choice Parameter from JSON file

EDIT: Updating my answer to include steps on using the Declarative Directive Generator.

You can experiment with the different available configuration options, and sometimes gets hints as to what certain settings provide. For this specific case:

Navigate to <$JENKINS_URL>/directive-generator/
Select parameters: Parameters from the Sample Directive drop-down.
Click the Add button and select extendedChoice: Extended Choice Parameter
Select JSON Parameter Type and then experiment with the available options.
When finished, click Generate Declarative Directive to generate the code block to paste into your pipeline.

You can add the javascriptFile parameter and change the type to PT_JSON.

Using your example:

properties([
    [$class: 'BuildDiscarderProperty', strategy: [$class: 'LogRotator', numToKeepStr: '500']],
    parameters([extendedChoice(
    defaultValue: 'One,Two,Three,Four',
    description: '',
    javascriptFile: '/path/to/file.json',
    multiSelectDelimiter: ',',
    name: 'SAMPLE_EXTENDED_CHOICE',
    quoteValue: false,
    saveJSONParameterToFile: false,
    type: 'PT_JSON',
    value:'One,Two,Three,Four,Five,Six,Seven,Eight,Nine,Ten',
    visibleItemCount: 10)


How should we organize VSTS repositories?

From the perspective of build automation, I find that it is better to have multiple repositories.  This allows for smaller configurations and more granular control of your build/release process.  You can allow references and pull in source code to build or release (of course you can also ignore source code as well in VSTS build pipeline).

This leads to being able to release just the project you want or need to your desired environment.  You can, of course, pull in artifacts from other builds and include them in your release if you need to.

To summarize, I tend to go as small as possible and only group things into the same repository if they are indeed part of the same project and no part of the project can be used elsewhere.


Should my zombie backend container be restarted when database is down?

The standard option is #2, keep retrying the connection, possibly with slower intervals, but I'd have an upper limit to the longest interval. The liveness probe should only fail if that specific container is broken, not any of it's dependencies. If there are cases where your application is failing and won't self recover when the DB is restored, then the liveness probe should fail in that scenario, and I'd consider an init container to check for the availability of the DB before starting the full application.

There should be some logging and metrics enabled to make the DB failure visible so that you debug the correct thing. If there is a concern of too many retries impacting the network, then a service mesh with circuit breaker functionality may be a good option.


How do I setup an OpenShift route that will allow both 80 and 443 traffic to my app?

You'll want to be sure your spec.tls section has the following two items to tell the router you want both secure and insecure traffic allowed in the edge.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: my-app-ui
  namespace: my-namespace
spec:
  host: my-host
  port:
    targetPort: http
  tls:
    insecureEdgeTerminationPolicy: Allow
    termination: edge
  to:
    kind: Service
    name: my-app-ui
    weight: 100
  wildcardPolicy: None


Is there a way to clone an ec2 from inside the ec2 terminal?

Yes, it can be done.

You can either create an IAM instance role which allows the instance to use ec2 actions such as "create-image" or you can create an IAM user and create for this user access and secret keys and a policy which allows this user to create images in ec2 and then configure the access and secret keys locally on the server using the "aws configure" command.

Whatever method you choose from the above options, the next step would be from within the instance itself and assuming you have aws-cli installed in the instance, first run:

ec2-metadata -i to get the instance id.

and then run:

aws ec2 create-image --instance-id INSTANCEID --name NameOfImage --no-reboot --region REGION

And it will create an image of this instance in the region you provided without rebooting the instance.

The create-image command will create an image of the instance, also called AMI which then you can use to create new instances from.


Multiple levels of parallel stages in a Jenkins declarative pipeline

Sure it's possible to have nested parallel stages and it works just fine (I've done it before), but the visualization in the web UI is terrible, even with Blue Ocean.  There was even an official blog post when this feature was added.

This question has already been asked a couple other times on this site as well, you might be interested in these questions and answers:

jenkins pipeline nested parallel
How to build a complex parallel Jenkins pipeline?

It's not entirely clear to me from the wording of your question if it's a duplicate of those questions or if there are significant differences, so I haven't flagged your question as a duplicate.


The --service-cluster-ip-range will define the range of IP address that Kubernetes will assign when you create a ClusterIP Service. It's different from your VPC CIDR which correspond to your nodes (EC2) IPs. You can let the default 10.32.0.0/24 as is or change it but make sure that it does not overlap with your VPC CIDR.


Which problems does a container orchestration solve?

If you are only running a single container or two containers together you are correct in that an orchestrator may be unnecessary and add unneeded complexity.  However, these tools do solve several issues when you running several containers together (especially in production).

  What are the common problems with containers that must be solved by these orchestration solutions

Containers run in an isolated process (usually in it's own namespace).  This means that by default the container will not be aware of other containers.  Additionally, it will not be aware of the systems files, network interfaces, and processes.  While this can greatly help with portability of the software it does not solve several production issues such as microservices, container discovery, scalability, disaster recovery, or upgrades.

  What is the benefit of adding such a complex system in your production?

Adding a container orchestrator can greatly reduce the complexity in production as these tools are designed to resolve the issues outlined above.  For example, Kubernetes is built to allow containers to be linked together, deploy containers across an entire network, scale and load balance the network based on container resource consumption, and allow upgrades of individual containers with no downtime.


How to add ECS attributes to an instance using terraform

Terraform docs are pretty comprehensive, so if it’s not in the docs it’s probably not implemented yet.

However, they’re very responsive to feature requests posted as GitHub issues, particularly those that indicate that the feature is available via the AWS CLI (or more specifically, the AWS Go SDK). In my experience, feature requests I’ve raised have usually been available in a new AWS provider release within a month or so.

I’d suggest doing up a new issue here:
https://github.com/terraform-providers/terraform-provider-aws/issues

(I did do a quick search to see if ECS attributes have been raised before and couldn’t see any at a quick glance, but you may like to as well!)


Terraform: can't connect to AWS provider using shared config file or static variable

The easiest way I've found to properly set credentials for my local runs and remote terraform cloud runs is to use environment variables. Before running set the environment variables for

AWS_ACCESS_KEY_ID
AWS_SECRET_KEY
AWS_DEFAULT_REGION

This should allow you to run in both Linux, PowerShell or even remote runs in Terraform cloud without issue.

I've gone between various approaches and this has provided me the easiest workflow. If you're doing this in AWS directly then of course you could use IAM instance role and avoid credentials altogether. This wouldn't allow you to keep the same workflow for local and remote though.


Docker compose scale, what is it good for?

Scaling is increasing the number of containers for any specific service. This is primarily helpful for a cluster, not on a single host setup like yours. When you scale your application up in a cluster, multiple containers of the same service get spread across the cluster (Based on their availability (Resource or anyway)). This makes the distribution of load across containers an available benefit. In a single host environment, with limited resources, scaling won't do you much good as the processes are going to be the slave of the same schedular, so you won't have that parallel servicing.
So now your question, why does scale in docker-compose exist? From the top of my head in no specific order of value:

You can minimize your services' downtime (going as low as none) while upgrading with docker-compose's scale functionality.
If an app can handle only so many inputs, in other words, if there's a queue of inputs, you can essentially increase the available queue space by scaling the container up and have multiple containers handle a portion of the inputs, reducing input loss.
The more obvious one, if one container goes down, there will still be at least one replica left.


How does one build a build pipeline with versioned build artifacts?

I know this question was asked a long time ago but I still want to answer it.
How products are build, versioned and delivered depend highly on the combination of the used tools, the infrastructure and the processes that have been established in your company.
This also means there is no general way to it. However there are of course best practices like continuous integration/continuous delivery, nightly builds, common branching patterns etc.
In your case I would start with common branching patterns for the used version control system (svn, git, etc.) and think about how to support versioned artifacts .
Next step is to get the build management tool (Maven, Gradle, Make, Cmake, etc.) build the specified version.
If these two parts are working you can think about how to automate it with jenkins.
One last thing: Your question was very general. I guess if you ask more specific questions like "How do I combine maven and git for building versioned artifacts" you get more answers.


How CircleCI, Terraform and AWS should interact in CI/CD Pipeline

There can be many correct answers, depending on the strategy you are doing. My take would be:

I wouldn't create a dedicated Docker image (e.g. index.docker.com/python:3.9-slim) with the application on it, but rather use the official Python Docker image, to build the app and the same to run it, so I'd package your app separately from the image that runs it. You can still compile your code as a Docker image if that's convenient and mount it as a volume. There are many benefits of this:

your code can be mounted as read-only, lowering the attack-surface
faster build (no need to add a full Python every time), faster deployment (yes, Docker layers would still work, but if your code changes, Python has updates, you might have multiple hosts eventually don't count on layering).
separation of concerns: just because you update your Python interpreter's patch version (e.g. for security fixes), your app don't have to change. This can be an important factor if you need to have a decision log for your releases.

You can use, e.g. CoreOS and use a Systemd unit file to pull down and run it for you, or you can instruct your Docker daemon via the Terraform provider to run it and set the "restart = always" so it will be started on reboots too. I'd go with the latter as it requires no knowledge on the OS and there is a single way (Terraform) to manage the whole deployment.

Well, if you have multiple instances, high-availability, then you either

need to do this in your app with a service-discovery tool (check
e.g. Hashicorp Consul) or you can have a load-balancer, but that
costs you money. Using load-balancers is the lazy way; using service
discovery and doing the load-balancing on the client-side (app)
offers you the best outcome, lesser architectural complexity, but
much more work.
Use an orchestrator, e.g. Kubernetes, but then your whole manual deployment with Docker and Terraform is not relevant anymore because you are dealing with the deployment at a higher level. Although Kubernetes has a steep learning curve initially, it is very straightforward when it is up and running.
A runner-up solution could be a DNS-based solution. In this case, you manually set, e.g. 3 instances and then in your DNS zone, you set a round-robin.
You can't use, e.g. AWS Autoscaling Groups, because you want to manage your containers via Terraform, so if the ASG scales up an instance, somehow your Terraform needs to run. So unless you start up your containers via "user-data", ASGs doesn't make sense, but in that case, you can't manage them via Terraform because Terraform is likely aware of the container's state. Even if you manage it, it will be a fragile architecture that will bite you back when you need that high availability.

Note: You probably need at least 2 Terraform run, because the Docker Terraform provider requires access to the target Docker service via TLS/SSH or Unix socket, so your instances have to be up and ready before you run the Docker one. But that's a good idea anyway because if you add all these in your user-data/cloud-init/Ignition script, Terraform will destroy the instances, so your container updates won't be that quick.
Note 2: Do yourself a favour and if you need more than 1 instances, use an orchestrator like Kubernetes on Hashicorp Nomad.


How to make different permissions for different subdirectories of storage bucket

Before giving the answer I'd like to clarify that S3 is meant for object storage, meaning that there's no file system involved and thus there are not "directories" per se.

So, an "object" with the name "/avatars/user-123/xyz" might give you the impression that it's stored in the "/avatars/user-123" directory, but there's no directory involved, that's just a name.

That being said, you can treat that portion of the name (/avatars/user-123) as a prefix, and then you can create IAM policies for your S3 buckets using said prefixes.

For futre reference, the following examples have been taken from:
https://aws.amazon.com/premiumsupport/knowledge-center/s3-folder-user-access/

In case readers find out that link has been taken down I'll just copy and paste the examples here

AllowStatement1 allows the user to list the buckets that belong to their AWS account. The user needs this permission to be able to navigate to the bucket using the console.

AllowStatement2A allows the user to list the folders within awsexamplebucket, which the user needs to be able to navigate to the folder using the console. The statement also allows the user to search on the prefix media/ using the console.

AllowStatement3 allows the user to list the contents within awsexamplebucket/media.

AllowStatement4A allows the user to download objects (s3:GetObject) from the folder awsexamplebucket/media.

{
 "Version":"2012-10-17",
 "Statement": [
   {
     "Sid": "AllowStatement1",
     "Action": ["s3:ListAllMyBuckets", "s3:GetBucketLocation"],
     "Effect": "Allow",
     "Resource": ["arn:aws:s3:::*"]
   },
  {
     "Sid": "AllowStatement2A",
     "Action": ["s3:ListBucket"],
     "Effect": "Allow",
     "Resource": ["arn:aws:s3:::awsexamplebucket"],
     "Condition":{"StringEquals":{"s3:prefix":["","media"]}}
    },
  {
     "Sid": "AllowStatement3",
     "Action": ["s3:ListBucket"],
     "Effect": "Allow",
     "Resource": ["arn:aws:s3:::awsexamplebucket"],
     "Condition":{"StringLike":{"s3:prefix":["media/*"]}}
    },
   {
     "Sid": "AllowStatement4A",
     "Effect": "Allow",
     "Action": ["s3:GetObject"],
     "Resource": ["arn:aws:s3:::awsexamplebucket/media/*"]
   }
 ]
}


Why does Helm time out on almost every action?

You have found a Kubernetes issue #22770, where there is a workaround mentioned here and it goes like follows:

What you're experiencing is a known issue with k8s where for some operations it expects to be able to resolve your node names in the global DNS.
And suggested a work around would be to:

Add entries to /etc/hosts on the master mapping your hostnames to their public IPs
Install dnsmasq on the master (e.g. apt install -y dnsmasq)
Kill the k8s api server container on master (kubelet will recreate it)
Then systemctl restart docker (or reboot the master) for it to pick up the /etc/resolv.conf changes

As last comment on the issue #22770 there is

I can confirm that adding the names of the our nodes to /etc/hosts on the kubernetes master resolves this problem for us too :)

, so there is not a fix in k8s coming in any upcoming version. At least from this particular ticket.


I've started abusing SyncThing for this purpose (among others), configuring it as a system service on laptops, then locking down it's UI to prevent it from being able to be used by a local connection originator to manipulate or access files. I get connectivity monitoring and robust rsync-like backup of field data.

It's fantastic at punching through difficult network conditions, so once you peer a node with a central node (say, a backup landing pad itself backed up via more conventional means), you can use Syncthing's API on your landing pad node to track connectivity.

If you're looking for end-to-end IP connectivity, and not just "can the remote host reach the Internet at all" knowledge, you can disable NAT traversal and relay use, at which point there will need to be a clean, unfirewalled path between the two nodes.

The downside is that SyncThing is very much a rapidly-developed project, with the issues that carries with it.


How to add label to Pull Request via Github API?

I made it work by changing the way labels argument is passed into requestBody.
Turns out the list representation while interpolated into string is missing quotation marks around each list element.
def setLabels (String repository, List label) {
    try {
        String labelsAsString = "[\"${label.join('", "')}\"]"
        def response = httpRequest httpMode: 'POST',
                        url: "[REPO_URL]/${repository}/issues/${CHANGE_ID}/labels",
                        authentication: 'ors_git_service_account',
                        requestBody: """{
                            "labels": ${labelsAsString}
                        }"""
    } catch (e) {
        wrap([$class: 'AnsiColorBuildWrapper', 'colorMapName': 'XTerm']) {
            echo "\u001B[31magdSlack.setLabels: '" + e.toString() + "'\u001B[0m"
        }
        throw e
    }
}


I implemented this in node.js - I don't know of a unix utility that can dynamically change grep. Here is how it works:

before dygrep:

many-child-procs > logfile.log                  # in terminal 1
tail -f logfile.log | grep <expression>          # in terminal 2

after dygrep:

many-child-procs | dygrep                    # in terminal 1

then you connect to dygrep with a tcp-client and control which expressions are filtered on.

The following node.js process reads from stdin, and uses a tcp connection to listen for expressions to filter on. For each line of stdin, it only forwards that to stdout if it matches one of the current regular expressions stored in memory.

#!/usr/bin/env node
'use strict';

import * as readline from 'readline';
import * as net from 'net';
import chalk from 'chalk';
import {createParser} from "./json-parser";
import log from './logger';

let port = 4900;

const rl = readline.createInterface({
  input: process.stdin.resume()
});

const regex = new Map<string, RegExp>();

rl.on('line', l => {

  if (regex.size < 1) {
    process.stdout.write(l + '\n');
    return;
  }

  for (let [k, v] of regex) {
    if (v.test(l)) {
      process.stdout.write(chalk.magenta(' (filtered) ') + l + '\n');
      break;
    }
  }

});

interface IncomingTCPMessage {
  command: {
    add: string,
    remove: string,
    list: boolean,
    removeall: boolean
  }
}

const server = net.createServer(s => {

  const sendMessage = (m: any) => {
    s.write(JSON.stringify({message: m}) + `\n`);
  };

  s.pipe(createParser()).on('data', (d: IncomingTCPMessage) => {

    if (!d.command) {
      log.error('No "command" field was found:', d);
      return ''
    }

    const c = d.command;

    if (c.list) {
      sendMessage({regexes: Array.from(regex.keys()).map(k => ({regex: regex.get(k), str: k}))});
      log.info('Listing all regex for the client.');
      return;
    }

    if (c.removeall) {
      regex.clear();
      sendMessage(`Cleared all regex.`);
      log.info('Cleared all regex.');
      return;
    }

    if (c.add) {
      regex.set(c.add, new RegExp(c.add));
      sendMessage(`Added regex: ${c.add}.`);
      log.info('Added regex:', c.add);
      return;
    }

    if (c.remove) {
      regex.delete(c.remove);
      sendMessage(`Deleted regex: ${c.remove}.`);
      log.info('Removed regex:', c.remove);
      return;
    }

    log.error('No matching field was found:', d);
    log.info('Regex:', regex);

  });

});

server.listen(port);

to see the complete code, see:
https://github.com/ORESoftware/dygrep


One could also use docker Jenkins LTS. If one wants to start from scratch, just remove the mounted Jenkins home folder and there will be a clean Jenkins in no time.

https://github.com/jenkinsci/docker/blob/master/README.md


Chef username/passwords best practice for multiple nodes

Storing sensitive data in attributes is definitely not a good practice. I once used a strategy of a chef-vault per environment (dev,stg,prod), and it worked well. If you use the sensitive data within templates you can set the sensitive property on the template resource to true to keep the password in plain text form from showing up in the run logs and associated template file diffs

We eventually moved to Hashicorp vault, and while it took more time upfront to learn and get up and running, it was definitely a better long term solution for secrets management and automation around managing sensitive data


Terraform | Error "transport is closing" when "terraform apply" after upgrading to 0.12

It's a bug in terraform aws ec2 module, last working version is 2.5.0


How can I access the console of a booting EC2 instance in AWS?

Background

When using cloud services such as AWS or Azure your ability to access the console of booting instances is typically restricted to a read-only type of interaction.

AWS does provide tools, so you just need to adjust your approach, if you're expecting to drop to single user mode or watch a booting system on a console.

For example, in the EC2 services area of AWS you can select a EC2 instance and then under the actions menu select either:

Get System Log
Get Instance Screenshot

Example

Step 1: Screenshot of actions menu

Step 2: Screenshot of "System Log"

Step 3: Screenshot of "instance's console"


Monitoring checklist - What things should I be monitoring?

I like this video: GOTO 2016 • Monitoring Microservices • Tom Wilkie

One of the key ideas (for me at least) is to realize the difference between host monitoring and application monitoring. Basically host monitoring tells you that something is fatally wrong now, but application monitoring should be able to predict problems by detecting higher error rate or that requests are taking longer time so you can fix problems before your users notice them.

(I'm not affiliated with weaveworks or the goto conference in any way, I just like the content and think there are some interesting ideas. Use the downvote button to let me know that this answer is not good :) )


Google Cloud SQL: Can I change machine type with zero downtime on my database clients?

I contacted the support of Google Cloud about this question and they told me that Cloud SQL isn't build to perform this change without downtime. If I want to be able to make these changes, I should look at Cloud Spanner which is a horizontal scalable SQL solution provided by Google.


Build image from inside a pod or container

If you simply need to build a docker image, I recommend using Kaniko. It allows you to build a docker image without having to expose the docker.sock or running your pods in privileged mode, which introduces potential security vulnerabilities. Per the Kaniko documentation, your spec file would look similar to:
apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=<path to Dockerfile within the build context>"
  restartPolicy: Never


Is there an established scheme for creating diagrams about Containers, Services etc.?

The established method is infrastructure as code.

Want to find out what processes run in a container? Check the Dockerfile. Infrastructure dependencies? Check the Cloudformation template/Terraform blueprint. Interactions between containers? Check the docker-compose.yml or Kubernetes yaml file.


How can I check for an installed packaged version on multiple linux hosts?

A simple way would be to do:

ansible -i <inventory> <target group, lets' say some CentOS systems> -m shell -a "rpm -qa | grep <package name>"

That would display a list consisting of server name and package version.


Is it an antipattern to avoid API Gateway for certain tasks?

You could have a separate API gateway for that kind of traffic.

Check out the "Variation: Backends for frontends" section of the following article:

https://microservices.io/patterns/apigateway.html


Continuous Integration and Isolated Feature Testing

First of all, git is certainly not to blame for merges taking a long time. Clean merging is the hallmark of git, unless it is used in some broken way. So I encourage you to look at what exactly seems to be the underlying problem. If your devs regularly make sweeping changes to the same files, then it's likely more a problem of cohesion/coupling than of the underlying merge tool.

Second; I like to do isolated feature-testing, if by that you mean that your test suite is ran against the feature branch (with only the branch changes) in a clean environment. In fact, I like to test all branches in this way. How could the develop or master branch become green if one of the features going into it are not? It goes without saying that develop / master need to be tested as well.

You could even go so far as to do a pre-merge test by doing a merge of feature into temp_develop_with_feature (or whatever you may call it), that is, simulate the merging of a branch into develop (under a different name) and test that.

Obviously, all of this assumes that sufficient testing ressources (VMs etc.) are available and fast enough, and that everything is 100% automated, both of which should be rather easy to achieve with something like Jenkins, Gitlab or whatever tool you use (if not, then it's maybe time to look at another one...).


Is the root user in a Docker image/container the same as the root user of the machine?

No the root in container is a root-user of container. Remember containers runs isolated.
DOcker uses cgroups and libvirt of kernel linux.


There are lots of good tutorials for NGINX and it is an easy server to setup with the Angular router.  You can still accomplish this with an Apache server, but you may need more steps in your Dockerfile. As a starting place check out the answer on Stack Overflow Deploy Angular 2 to Apache Server.  These types of server modifications will need to be baked into your Dockerfile.


Using the built binary of a docker image in another docker image

What you need is to use the build pattern or the multistage features of docker

https://docs.docker.com/develop/develop-images/multistage-build/

With that, you can use as many images as you need depending on your build process


What is a good storage for storing performance results and visualizing in grafana

Sadly nobody answered, so I will provide feedback about how i did it so far.

Graphite was not a good solution for me due to its retention policy. Performance tests sent data to the database every millisecond, and due to retention Graphite did not store all the results but only the last one per the period, meaning if you sent 100 data points in the last minute, the first 99 would be dropped, the 100th one would be stored. This fact was found by me somewhere on some forum. I do not claim I have setup/configured/used graphite correctly, yet the result was several days of debugging a bug which appeared an actual feature, lots of stress and questioning if my head and hands grow out of my shoulders.

Next option was to use InfluxDB. Sadly didn't work for the following reason:
 - If running InfluxDB in a Linux docker container, I was not able to bind mount it to a specific folder on host's OS outside of docker installation folder.
A solution would be to try and use docker named volume. Maybe that would have worked, but I was short on time and patience.

The final solution was to use simply a Postgres DB. Created a timetable with all the metrics and a timestamp field. As a result, Locust writes the final session statistics into a CSV file. CSV file is parsed by a python script and parsed data is inserted into the table. Grafana then connects to the DB and quite nicely visualizes the results using a simple query.

Next possible question would be:

  how to store the results of every single request to gain better
  precision for the results of my performance tests?

My personal suggestion would be to use influxDB with a consistent storage saved in a named volume if you are using Windows host. There the smallest time period can go down to milliseconds and data retrieval was very easily setup by manually writing a simple select query. Yet, please do not take my word for it and try this out, because i haven't done this, so my thoughts are nothing but a theory.

Happy performance testing.


How to determine if a Jenkins build is triggered by a Pull Request merged to Master

You might try installing the GitHub Integration plugin. Reading their documentation, they provide a number of environment variables you can use for your purpose.

GITHUB_PR_STATE can be OPEN, CLOSE
GITHUB_PR_SOURCE_BRANCH for the source branch (e.g., hotfix/foo)
GITHUB_PR_TARGET_BRANCH or master

Using the when condition, you can run a job/stage when it's a closed PR merging into the master branch using:

when {
  allOf {
    expression { env.GITHUB_PR_STATE == "CLOSE" }
    expression { env.GITHUB_PR_TARGET_BRANCH == "master" }
    expression { env.GITHUB_PR_SOURCE_BRANCH == "hotfix/foo" }
  }
}


Assuming the following:

YOUR_PC, has Redis client, SSH client, SSH access to SSH_SERVER
SSH_SERVER, has SSH server, redis access to REDIS_SERVER
REDIS_SERVER, has Redis server

Set up the tunnel from YOUR_PC, port 1234, to REDIS_SERVER:REDIS_PORT, via SSH_SERVER

ssh SSH_SERVER -L 1234:REDIS_SERVER:REDIS_PORT

On another terminal, on YOUR_PC, run the redis client (based on https://stackoverflow.com/a/40678950):

redis-cli -h localhost -p 1234


Which SCM solutions exist for managing mainframe software?

From my own experience, these are some the typical softwares packages:

CA Endevor.
SERENA ChangeMan ZMF.
Compuware ISPW.
IBM SCLM.

All of these packages can manage, more or less out-of-the-box, anything that's stored in regular "PDS" components (a typical file structure used in z/OS).

When it comes to a company evaluating which one best fits for them, it often boils down to these criteria:

IBM SCLM is perceived as free of any license/maintenance fees (actually it's included in the z/OS license, which itself is not free). So if no dedicated budget is available, then this is often the software package that gets selected (better then no package at all). If there is a budget, then this one is often the one that doesn't make it to the shortlist.
CA Endevor has the highest install base. Its key strength, IMO, is the way that you can trace down for each executable how it has been compiled/linked using which version of which building blocks (copybooks, etc).
SERENA ChangeMan ZMF's install base is quite a bit below the one from CA Endevor. Some of its key strengths are:

the notion of "packaging" related software changes, which is at the core of it.
its capabilites to deploy softwares towards physically remote sites.

Compuware ISPW is like the "new kid in town" (as compared to the CA Endevor or SERENA ChangeMan ZMF alternative). It is commonly perceived as the solution where "any custom SCM requirements can be implemented with it, with a relatively low effort to do so".

Looking at it from an architecture standpoint, then SERENA ChangeMan ZMF and Compuware ISPW appear to have the most open architecture, which is what you'll need if you want to tune it to make it manage software components written in a 4GL language that is (what some call) more exotic, as compared to 3GL languages such as COBOL, PL/I, etc. I.e. because the software components are stored in file systems that are not stored in standard PDSs. Some examples of those languages are:

Natural (stored in ADABAS DBMS).
ADS Online (stored in IDMS DBMS).
QMF (stored in DB2 DBMS).
CA Gen (stored in "encyclopedias").
CSP (stored in VSAM file format).

Attention: having an "open architecture" is great to have it fit your custom requirements (the sky is the limit). However when it comes to upgrading to new releases, it comes with a price to get those custom requirements upgraded also.

Note: rather accidentally, during a CA Endevor training for SERENA ChangeMan ZMF experts, we discovered that CA Endevor and SERENA ChangeMan ZMF appear to have the same roots (from somewhere in the late 1980s ...). For those who are a bit familiar with both of them: go check what the functionality is of these utility programs, with similarly named names ... (you'll be shocked ...):

PGM=CONWRITE versus PGM=CMNWRITE.
PGM=CONPRINT versus PGM=SERPRINT.


Set a Jenkins job to not to clone the repo in SCM

Yes, definitely. I do this all the time. You can specify configuration options for your pipeline and one of them is skipDefaultCheckout, which causes pipeline to skip the default "Declarative: Checkout SCM" stage.

The skipDefaultCheckout option is documented in Pipeline Syntax and here's an example Jenkinsfile showing how to use it:

pipeline {
  agent { label 'docker' }
  options {
    skipDefaultCheckout true
  }
  stages {
    stage('commit_stage') {
      steps {
        echo 'sweet stuff here'
      }
    }
  }
}


Kubernetes - unable to login to Dashboard

how we can extract the certificates from the kubeconfig file:

Locate your kubeconfig or config file which you use to run kubectl commands. If you have used my Vagrant file above, you can find it on /etc/kubernetes/admin.conf
You need to export a single file (.p12) with the following two certificates: the client-certificate-data, and the client-key-data. My example runs the command on /home/vagrant. If you run this command on macOS, be sure to change the base64 -d to base64 -D

$ grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt

$ grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key

$ openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12

Import the kubecfg.p12 certificate, reopen your browser, and visit the Kubernetes Dashboard URL. Accept any warning and you should see the authentication page. You can skip the login and check you are not able to perform any task.
The following steps have been copied from the Kubernetes Dashboard wiki page (Creating-sample-user)

    1- Create service account

        cat <<EOF | kubectl create -f -
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: admin-user
          namespace: kube-system
        EOF
        >>

    2- Create ClusterRoleBinding

        cat <<EOF | kubectl create -f -
        apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRoleBinding
        metadata:
          name: admin-user
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
          name: cluster-admin
        subjects:
        - kind: ServiceAccount
          name: admin-user
          namespace: kube-system
        EOF
        >>

    3- Get the Bearer Token. Once you run the following command, copy the token value which you will use on the following step.

        $ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

    4- Come back to your browser and choose token on the login page. You will need to paste the token value you have copied on the previous step.
5- Click “SIGN IN” and you should be able to see your Kubernetes Dashboard fully operational.


Is there a Jenkins plugin for downloading source code?

Perhaps you are looking for the Job Cacher plugin?
To quote the plugin page:

Summary
Enables caching of files on transient executors for use on subsequent builds to improve performance
Features

Item storage extension point supporting on master storage and AWS S3
Cache Wrapper for free style jobs that manages the cache
Arbitrary File Cache where user specifies paths to be cached
UI on Job page to review the job's caches
Supports Pipeline jobs with a cache directive
Cache Extension Point for other plugins to provide opinionated caching capability such as Gradle caches
Incremental storing of cache into storage to minimize data movement

However, these are for files created by the jobs, not needed by them. In order to use it in your case, you might need a few dummy jobs specifically for downloading the content from the arbitrary URLs, and then caching them in your Jenkins cache. I guess you'd have to write the script for that.
To be frank, I've never used this plugin, and what I describe above doesn't seem to be an elegant solution. Your question specifically ask for a Jenkins plugin, however I believe the general problem is how to cache job dependencies in Jenkins. If you agree to this wider interpretation, there are some other options.
I have often solved the problem you mention using a combination of managed scripts and persistent storage which Jenkins and workers have access to, in Jenkins.  If you need really arbitrary dependency management, a tool like Spack or EasyBuild may be useful. Although these have evolved in the HPC domain, they do quite well at caching dependencies for workloads.
However, since you mention just a few dependencies (specific sqlite versions e.g.), I think it may be worth your time to write a script which does the caching for you and, together with globally-available persistent storage, do the caching and retrieval that way.


Using a combination of Terraform and Ansible to create dynamic AWS EC2 infrastructure

There are a lot of gaps in your deployment and configuration strategy. In order to provide a more robust and elegant strategy, I suggest you:

Define autoscaling groups with Terraform instead of provisioning individual EC2 instances. Auto-scaling groups make dynamically scaling instances a much simpler process. It will also allow you to define scaling policies and health checks for your instances to determine when to add/remove instances from your autoscaling group.
Create and define CloudWatch Metrics and CloudWatch Alarms. CloudWatch allows you to create custom metrics to monitor and alert via CloudWatch Alarms. This can be used in conjunction with autoscaling group scaling policies to define a threshold to scale your instances based on metrics such as cpu utilization, application response time, etc.
Instead of using Ansible to configure each EC2 instance, use Ansible to configure an Amazon Machine Image (AMI) to be uploaded and used by your autoscaling group. If your instances are configured identically, configuration is a one and done process. If your instances are configured differently, consider uploading multiple AMIs, and separating into multiple autoscaling groups. You can also use the User Data definition of an EC2 instance to run shell commands on each individual instance at launch (a one time process).
Last, but not least, instead of running this process from "a devops machine", consider using a build/deploy CI pipeline to run and configure this entire process. Tools such as AWS CodePipeline, Jenkins, GitLab CI, etc. allow for you and other team members to run your configuration in a configured and repeatable manner. It will also allow you define stages and deployment strategies should you choose to implement testing for your application later on in your development process.


How do you manage your inventory of spot instances on AWS with Ansible

You are looking for Dynamic Inventory

Please look into ec2.py and ec2.ini of Ansible, and also the Dynamic Inventory docs of Ansible are very good for understanding how to manage your AWS inventory.


AWS or Azure with usage auto shutdown occasional usage

I can think of a few different ways to achieve this.

First and easiest way is to create a new CloudWatch Events Rule 1, set its "Event Source" to run on a schedule and "Target" to EC2 StopInstances API call with Instance IDs provided. Look at this screenshot for clarity:

With this example, An event will start at every night at midnight and it will stop Instances by their IDs. You can change the schedule with cron expressions. See: https://crontab.guru

You can implement any kind of custom logic with AWS Lambda functions and schedule them to be run with a variety of AWS services (including CloudWatch Events).

For example, with a Lambda function you can implement logic to find instances by a tag, like "GoodBefore=2018-08-10", and stop these instances. In fact you could even make use of API Gateway to expose an API for requesting instances, only to be terminated after a few days by their "GoodBefore" tag.

Of course, while this way is much more customizable and sophisticated, it may prove hard especially if you don't have experience in programming.

If you don't want to implement custom logic with Lambda, but also need more than what CloudWatch Events can provide, you can make use of Instance Scheduler solution provided by AWS. It's a similar solution to what I described in previous method, but this time AWS already did the hard work for you.

1 URL for us-east-1 region: https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#rules:action=create

Note: While I haven't used Instance Scheduler myself, I'm pretty sure it wouldn't decide on such things, and I think it only makes sense if it wouldn't do much more than simply making a stop call for the instance. So, on any stop event, either made by an automated solution or you, if your instances' root volume was created "EBS-backed", it will retain the data. If it is "instance store backed", it will lose the data. You can find this information from AWS EC2 console. If you want to retain data after stopping instances, make sure all volumes are EBS-backed when creating a new instance.


Private docker registry that attempts a pull-through on images not found locally

Yes, it is possible. The entire subject is discussed in Registry as a pull through cache, from where the below quotes are taken.

Basically you need to configure the cache as a proxy for Docker Hub:

  Configure the cache

  To configure a Registry to run as a pull through cache, the addition
  of a proxy section is required to the config file.

  In order to access private images on the Docker Hub, a username and
  password can be supplied.

proxy:
  remoteurl: https://registry-1.docker.io
  username: [username]
  password: [password]

Of course, you also need to configure your docker daemon to use your local cache (but I presume that may be already done from the context of the question):

  Configure the Docker daemon

  Either pass the --registry-mirror option when starting dockerd
  manually, or edit /etc/docker/daemon.json and add the
  registry-mirrors key and value, to make the change persistent.

{
  "registry-mirrors": ["https://<my-docker-mirror-host>"]
}


How to access report data from Sonarcloud.io aka SonarQube API, or functionality no more available?

Ok no so obvious but at least one possible way is:

Note the component ID in the TravisCI report
Use the ID and this documentation to get data:

https://sonarcloud.io/api/measures/component?componentId=&metricKeys=code_smells