Why are IaaS hosting services are usually fixed-priced and PaaS are usually dynamically-priced?

Fixed price tiered billing is an older billing model that suits a monthly billing cycle where usually the level of resources provided does not change over a month. Even if it is changed, this is usually met with friction from the billing department in form of the higher pricing tier being applied to the whole month.

By comparison AWS can be credited as the first provider to offer simple building blocks on an hourly basis (now per second). This allows for much more sophisticated usage of resources in form of:

auto scaling to the best amount of resources required.
turn services off out of hours or until turned on manually.
ultimately to offering functions as a service, the most finegrainerld resource billing usage available.

So I don't see these two items as being directly comparable, but one is a progression from the other.


Unable to create container after installing mysql and phpmyadmin | Ubuntu

Pulling the images only make them available for you to use locally. Therefore, to use them within a container you should run the command docker run with the appropriate configuration for each services


I looked at Jenkins plugins for vault, but they only work for fetching secrets from Vault.

It depends how the current configuration looks like that is used to deploy apps.

https://github.com/jenkinsci/hashicorp-vault-plugin

If one uses Jenkins pipelines, then one could replace the keyId with the one that is defined in Hashicorp vault.

node {
  // define the secrets and the env variables
  def secrets = [
      [$class: 'VaultSecret', path: 'secret/testing', secretValues: [
          [$class: 'VaultSecretValue', envVar: 'testing', vaultKey: 'value_one'],
          [$class: 'VaultSecretValue', envVar: 'testing_again', vaultKey: 'value_two']]],
      [$class: 'VaultSecret', path: 'secret/another_test', secretValues: [
          [$class: 'VaultSecretValue', envVar: 'another_test', vaultKey: 'value']]]
  ]

  // optional configuration, if you do not provide this the next higher configuration
  // (e.g. folder or global) will be used
  def configuration = [$class: 'VaultConfiguration',
                       vaultUrl: 'http://my-very-other-vault-url.com',
                       vaultCredentialId: 'my-vault-cred-id']

  // inside this block your credentials will be available as env variables
  wrap([$class: 'VaultBuildWrapper', configuration: configuration, vaultSecrets: secrets]) {
      sh 'echo $testing'
      sh 'echo $testing_again'
      sh 'echo $another_test'
  }
}


Ansible fetch all vars files from a directory to generate a combined vars file

Sounds like you want the combine filter.

You could read each file into a specific variable, then use the combine filter to create all-pets. At that point, it would be in memory of course, so you might want to use the answer to this question to write it to a file:

- name: Read vars
  include_vars:
    file: "pets/{{ item }}_pets.yml"
    var: "{{ item }}"
  loop:
    - steves
    - jacks

- name: Combine vars
  set_fact:
    all_pets: {{ steves | combine(jacks) }}

- name: Write the file
  copy:
    content: "{{ all_pets | to_yaml }}"
    dest: pets/all_pets.yml


CORS is applied at the application layer, not the infrastructure (docker) layer. So, this has nothing to do with docker. It also has nothing to do with the axios client. Rather, your server needs to allow requests from your client by returning the correct headers via the OPTIONS method.
If you're using nodejs and express, you can add something like this to your service:
app.use((req, res, next) => {
  res.header("Access-Control-Allow-Origin", "*");
  res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept, Authorization");
  res.header("Access-Control-Allow-Methods", "GET");

  if (req.method === "OPTIONS") return res.status(200).json({});

  next();
});

Or, there are plenty of CORS libraries that can help you out.


How to add my own columns to Jenkin's Dashboard

you can use the Jenkins Dashboard View to add the layout of your choice. The page says:

  The dashboard view comes with a number of portlets that you can configure your view with

so you can add top, bottom, or column portlets as you seee fit.


Customizing deploy key and ssh wrapper path for git cloning through application_git cookbook

Quoting from the cookbook README:

  deploy_key – SSH key to use with git. Can be specified either as a
  path to key file already created or as a string value containing the
  key directly.

And:

  The application_git resource deploys code from git. It extends the
  core git resource to support deploy keys and disabling strict host key
  verification.

So checking the core git resource documentation:

  ssh_wrapper Ruby Type: String

  The path to the wrapper script used when running SSH with git. The
  GIT_SSH environment variable is set to this.

so something like this (untested) should do (using the wrapper in the cookbook) :

cookbook_file '/path/to/my/desired/ssh_wrapper.sh' do
  source 'my_ssh_wrapper.sh'
end

application 'accounts' do
  action :deploy
  path <somepath>
  owner 'nobody'
  group 'nogroup'
  git 'name' do
    repository <repopath>
    revision <value>
    ssh_wrapper '/path/to/my/desired/ssh_wrapper.sh'
    deploy_key deploy_key
 end
end

You can of course do the same with the deploy_key, but using something like chef_vault or an encrypted databag sounds better to avoid commiting deploy keys within the cookbook.


How to check AWS spot pricing automatically?

Spotted an open source tool called autospotting that just might help with this:

  Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.

We have this in our to-do pipeline, will be able to add more context once we finish that.

Update:

Another tool that was recently demo'ed in a conference was mapbox/spotswap

This works slightly differently. It monitors a normal ASG with either On-Demand or Reserved instances and then, if scale arises, bids and provisions spot instances of similar compute level on a separate ASG.


Where to obtain the token to register a GitLab runner?

For a specific GitLab runner tied to a project, the token can be found in Step 3; in Project -> Settings -> CI/CD -> Runners in the Setup a specific Runner manually section.

You will also be able to find the URL which you need to use to register the runner in Step 2.


AllowedPattern for emails in Cloudformation template

Important
Since AWS CloudFormation templates use the JSON syntax for specifying objects and data, you will need to add an additional backslash to any backslash characters in your regular expression, or JSON will interpret these as escape characters.
For example, if you include a \d in your regular expression to match a digit character, you will need to write it as \\d in your JSON template.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-regexes.html
The very same applies to your \s expressions. Use \\s and it works.
EDIT:
Looking back the question I started to wonder the missing quotation marks on your definition. I found out the expessions are yaml format that is the second way to define things in aws.
If you should follow the yaml syntax, the right way to give string values is like this:
AllowedPattern = '/[^\s@]+@[^\s@]+\.[^\s@]+/'

https://aws.amazon.com/blogs/aws/aws-cloudformation-update-yaml-cross-stack-references-simplified-substitution/
EDIT:

Use quotes if your value includes special characters, (e.g. :, {, }, [, ], ,, &, *, #, ?, |, -, <, >, =, !, %, @, ).

https://stackoverflow.com/questions/19109912/do-i-need-quotes-for-strings-in-yaml


Running two tillers does not make it HA. There is an open issue at the moment.


Spot Fleet aims to meet your capacity requirements across a range of instances. This reduces the risk of any unavailability of a specific instance type or family affecting your workload.

As you are interested in only one specific machine type, you will need to remove the other types from your spot fleet request; or just create a separate spot instance request for the p3.2xlarge (a plain spot instance request, not a spot fleet request)


Relation/interplay of deployment pipeline stages (dev-stage-prod) with blue/green deployment mechanisms?

Yes.  With no QA and wanting to have an internal environment separated from production this makes sense.  However, a couple comments:

You should go for whatever is going to make your business money instead of going for an enterprise standard.  While some of the common practices may not be sensible yet, keep in mind that as your company grows (e.g. a dedicated QA role), they might eventually be very useful.
If you get rid of any stages, you are essentially eliminating a development environment.  Straight to production can be great, or greatly harmful.


Anyone know how to use $CIRCLE_BUILD_NUM across different jobs in CircleCI?

This is the approach I came up with.  It's kinda ugly, but it does work:

- run:
    name: Create unique build identifier
    command: |
        GIT_SHA=$( git rev-parse --short HEAD )
        echo "export GIT_SHA=${GIT_SHA}" >> "${BASH_ENV}"

        # the always-increasing counter, based on CIRCLE_BUILD_NUM
        BUILD_COUNTER="${CIRCLE_BUILD_NUM}"
        echo "export BUILD_COUNTER=${BUILD_COUNTER}" >> "${BASH_ENV}"

        # the build identifier, which includes the short git sha
        BUILD_NUMBER="CIRC${BUILD_COUNTER}-${GIT_SHA}"
        echo "export BUILD_NUMBER=${BUILD_NUMBER}" >> "${BASH_ENV}"

        # output build id and counter
        echo -e "\nbuild counter: ${BUILD_COUNTER}; build id: ${BUILD_NUMBER}\n"

Then ${BUILD_NUMBER} is available as an environment variable in other steps.


How can I determine which command / step failed within a travis build

Look at the Travis CI Logs.  Often a failed build step will return "Error", "Failure", "-1", buried in a wall of text.  If you want a better way to collect and  these logs you can check out the following:

Official Github page with details on how log collection works
An official blog on how to structure your production logs.


I tried the dumbest approach, i.e. just to plug the iso.xz file into iso_url and see what happens. Of course, this failed (even after manipulating the iso extension key).

What I ended up doing --- knowing a lot about Bash scripting, and not much about Packer, Go and the like --- is to wrap packer calls in a script that downloads and decompresses the image, then calls Packer on the decompressed image.

I would be curious to know if there are more elegant approaches.


How to avoid "retry storms" in distributed services?

It depends on what you're trying to avoid.

If you are trying to avoid any service interuption of something which is a genuinely critical service (I'm thinking in terms of "people will die if my API call is not appropriately served") the you need to just budget for the huge inefficiencies that come from vastly over provisioning dedicated resources. And yes they have to be dedicated, none of this allowing for traffic spikes stuff, multiple services spiking would thus cause an outage.

In the far more likely scenario that you're service going down would be inconvenient you can tackle the problem both from the client and server sides. Although it's worth noting that it's logically impossible to actually solve the problem of to much traffic because without processing the traffic (which consumes resources) you can't know if it's a retry, if it's a retry for a request that was successful but incorrectly handled by the client, if it's a DDOS, etc. But you can mitigate impact.

In the client code write sensible retry logic which has an upper limit and a mechanism for gracefully failing. That way you don't stick your users in an infinite loop of failing requests and you just give them an error telling them to try whatever they just did in little while.

For your server side infrastructure the simplest solution is to throttle. Hard limits on requests, especially if you can try and spread them logically based on your specific use case (ie. If you have a centralised service make some hard decisions, do you want to start blocking geographically distant requests which might be resulting in threads hanging server side? Or do you want to distribute your inevitable yet minor outage evenly? etc) It basically boils down to the fact that returning a 503 intentionally from a gateway is a hell of a lot cheaper than letting the request go through and sending a 504 anyway. Basically force clients to behave based on what you can currently provide and provide the correct responses so that clients can react appropriately.


Module directory modules/queues.tf does not exist or cannot be read

A Terraform module is represented by a whole directory, not an individual file. If you create a directory ./modules/queues and move ./modules/queues.tf to ./modules/queues/queues.tf then you can call that module like this:

module "pae_queues" {
  source = "./modules/queues"

  environments = var.environments
}


Tunnel docker traffic though VPN or SSH, during build stage

Docker actually manipulates iptables behind the scenes, so there is a good chance that sshuttle is interacting with docker in unpredictable ways.  Any rules that need to be added to IP Tables that affect the docker interface need to be added to the DOCKER-USER chain as per Docker and iptables.

It's quite an involved topic so I would also consider reading an exploration of Docker networking and the overview of the docker network.

There is also an old question on StackOverflow but based upon the comments the current highest voted answer no longer works.


DevOps tools and automation of manual processes

Alright... you're not going to implement every tool or automation at the same time, as some of them have quite deep impact on your development processes (and I daresay, development culture). Take a step-by-step approach; research each individual tool. Figure out what it actually does, what it is useful for. Play around with them, install them locally, go through tutorials, and see what they do, or whip up a quick VM somewhere (AWS, etc.) if you need more than one.

Then, take a look at the work your team/company is doing right now. What are the manual process you wish to automate? Ideas should spring up pretty quickly.

A nice, simple(-ish), bog-standard stack to start out with:

Vagrant to automate setting up VMs for your developers, or test environments.
Ansible to automate configuration management (relatively lightweight compared with Chef or Puppet)
Gitlab or Jenkins as your automated CI/CD driver, mostly depending on taste
Docker for your containers
docker-compose, docker-swarm, Kubernetes, OpenShift to automate container deployment
ELK or Splunk for your logging/monitoring, again depending on taste

Have fun!


The "how" of extremely high-frequency deployment is all about very mature engineering practices and tooling.  The "unicorns" (Gene Kim's term for the type of company you list) have some of the best talent in the industry, and invest in the best technology.

One of several key practical differences between these unicorns and most other orgs is the level of automation they employ.  It is very difficult to automate the quality checks needed to maintain a constant flow of work from DEV, through testing environments, to Production.  Development and Operations must maintain high service levels internally.  Ops must provide self-service resources so developers aren't waiting on tickets to get new infrastructure.


Logging & Monitoring on multiple servers

Well, I spent hours reading lots of articles, documentation, and finally figured everything out. See the comparative table:

Brief description
Zabbix is a full-fledged technology that can deal with both metrics and logs processing, yet can be extended. It also supports automatic hosts detection & network maps building.
Prometheus is a metrics monitoring system. It is Kubernetes native, thus it can automatically detect hosts for data extracting. It also has myriads of available adapters: docs: Exporters And Integrations. To visualize the data, Grafana is used. Speaking about data storage, Prometheus has its own local system for that, but there are integrations with other technologies for sharding, see docs: Storage.
PLG is a stack for log aggregation. Logs get collected with PromTail, Loki is a core processing engine, and Grafana is used as an interface. Loki has no own storage system, so some other solution is required. It stores data as chunks and indexes, which are usually stored separately docs: Storage. As I understood, PLG has no trigger/notification system, but the data extracting part also can parse logs into metrics and export them into Prometheus, so we get alerts. Also, I met a notice that PromTail doesn't support multiline logs, but FluentD can be used instead as it is more flexible.
ELK / EFK is a technology stack for logs parsing, indexing, searching, so it can perform complex queries. For data extracting, LogStash was originally used, but FluentD is becoming popular.


What is the best way to share maven repository using Docker build (dynamic) image with maven?

You might use docker builder pattern. Briefly you need to create Dockerfile.build which adds pom.xml and run mvn dependency:resolve:

FROM maven:latest
ADD ./pom.xml /src/pom.xml
WORKDIR /src
RUN mvn dependency:resolve

Rebuild that image every time prior build. Docker will use cached image if ./pom.xml has not changed.

Docker 17.05 introduced 'multi-stage builds' feature which implements this pattern. More information is available here: https://docs.docker.com/engine/userguide/eng-image/multistage-build/


Is it possible to deploy RDS in multiple region in AWS with read and write replica

You can look at Aurora Global Database for this. Based on this news announcement, you can also configure the setup, so you can write to your local db cluster and this is forwarded to your primary region.
Of course there would be some lag in write requests, essentially of the length of time it takes to write to the primary region and replicate the data changes back to the replica in your local region. Nonetheless this is likely easier to handle (and with the same methods in a web application), as if you wrote to the primary region yourself and checked if the data had arrived back in local replica.


Extra none:none images after docker build

Those images are the docker build cache. You need this to be able to quickly rebuild similar images, and to avoid rebuilding unchanged layers which would result in resending those to remote registry servers and forcing servers to download those unchanged layers again. One important point on these layers:

They do not consume any significant disk space

You can view the layers of an image by running the docker image history command on an image. Here's an example of a rather large image built on my machine:

$ docker image history 25f90c8ba1ca
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
25f90c8ba1ca        11 days ago         /bin/sh -c #(nop)  CMD ["/bin/sh" "-c" "echo…   0B
8a190467427e        11 days ago         /bin/sh -c #(nop) COPY dir:36bb9e67a0f2844c2…   298MB
1f6f5474071d        11 days ago         /bin/sh -c #(nop) COPY dir:7c364877951bb2312…   249MB
007510a95d17        11 days ago         /bin/sh -c #(nop) COPY dir:9817b6e3d212385d0…   231MB
ee23b7a42488        11 days ago         /bin/sh -c #(nop) COPY dir:57e817f3d9d977f66…   277MB
e190d1bb3f66        11 days ago         /bin/sh -c #(nop) COPY dir:9ba997e629e74ca64…   205MB
abe0ec5c0ef9        11 days ago         /bin/sh -c #(nop) COPY dir:4dbbd5b2010590e28…   213MB
544af937d8a3        11 days ago         /bin/sh -c #(nop) COPY dir:0ac40edba2c7e1317…   178MB
b78085aba431        11 days ago         /bin/sh -c #(nop) COPY dir:2137a075c552d6783…   135MB
3ff8488b067b        11 days ago         /bin/sh -c #(nop) COPY dir:45da2005488e81e9c…   112MB
19b7cdd9d682        11 days ago         /bin/sh -c #(nop) COPY dir:2b745a4311740ddf7…   65.5MB
eb2cf84859fb        11 days ago         /bin/sh -c #(nop) COPY dir:cd7b639f2eee0e9da…   50.1MB
1ba1c21bb0a3        11 days ago         /bin/sh -c #(nop) COPY dir:594739607d1f2d593…   51.9MB
1da5d6bf03e9        11 days ago         /bin/sh -c #(nop) COPY dir:757daf68a395cfaae…   30MB
f6df8b695822        11 days ago         /bin/sh -c #(nop) COPY dir:449b6c2ff93606741…   18.5MB
72895d213dac        11 days ago         /bin/sh -c #(nop) COPY dir:2290eb8d11f15dbe6…   13.8MB
3b6ceadce77b        11 days ago         /bin/sh -c #(nop) COPY file:fcd71a6bfc87e9dd…   167B
52dd0e701be3        11 days ago         /bin/sh -c #(nop)  ENV TARGET=/usr/share/ngi…   0B
aae476eee77d        3 weeks ago         /bin/sh -c #(nop)  CMD ["nginx" "-g" "daemon…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  STOPSIGNAL [SIGTERM]         0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  EXPOSE 80/tcp                0B
<missing>           3 weeks ago         /bin/sh -c #(nop) COPY file:1d1ac3b9a14c94a7…   1.09kB
<missing>           3 weeks ago         /bin/sh -c #(nop) COPY file:af94db45bb7e4b8f…   643B
<missing>           3 weeks ago         /bin/sh -c GPG_KEYS=B0F4253373F8F6F510D42178…   13.3MB
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV NGINX_VERSION=1.15.5     0B
<missing>           4 weeks ago         /bin/sh -c #(nop)  LABEL maintainer=NGINX Do…   0B
<missing>           6 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           6 weeks ago         /bin/sh -c #(nop) ADD file:25c10b1d1b41d46a1…   4.41MB

$ docker image inspect --format '{{json .RootFS.Layers}}' 25f90c8ba1ca | jq
.
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e",
  "sha256:5fcfba1ba609a8fbfc3eccbc805eb18a78041897ec3f6dc168c9ed871baed950"
]

$ docker image inspect --format '{{json .RootFS.Layers}}' 8a190467427e | jq .
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e",
  "sha256:5fcfba1ba609a8fbfc3eccbc805eb18a78041897ec3f6dc168c9ed871baed950"
]

$ docker image inspect --format '{{json .RootFS.Layers}}' 1f6f5474071d | jq .
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e"
]

In each of those inspect commands you should see that each new image is just adding on an extra layer until you reach the final image. These layers are the same sha256 checksums, pointing to the same files on the disk. Docker does not copy these files per image, it doesn't even copy them when creating containers (excluding initializing named volumes), the layers are read-only and reused by every descendant image and container via a union filesystem. The disk usage that docker displays is the sum of all of the layers even when other images may share layers, while your actual disk usage is only of the unique layers.

If you rebuild images often on the server with the same build tag, you can prune dangling images with docker image prune. Otherwise, simply avoid displaying these build artifacts by running docker image ls without the -a option.


How to create staging servers for multiple git branches?

1) How can I run several different branches on the staging server?

Docker

  2) How would I set up the DB evolution system to make sure it always has appropriate DB for each branch?

This depends on how much you expect your DB to scale. You can get pretty crazy with methods to clone database data but typically you will want a master copy that you do not alter until your code is released into production - and keep good backups when you do. While you infrastructure can be immutable and disposable, your data is not. You can only emulate disposability with you data. 4.2 GB really isn't that much to copy. You could make a script to spawn a new database instance for each build to operate on, then discard it once testing is complete.

  3) How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?

You could consider using something like git hooks to trigger a build, force a code checkout or begin spawning a container and cloning your database. You could make an API call to a build automation system like Jenkins and/or use it to kick off a configuration management system like Puppet, Chef, Salt Stack, Ansible or something else. This would be more like an auto-pull than an auto push.

From the wording of your question, it's clear you are thinking of a mutable infrastructure, but consider using immutable testing deployments instead.


What are the responsibilities of a project manager in a DevOps project?

Truth would be that should not happen in a DevOps environment
.
Your "project" will probably use Agile concepts, but there's nothing in DevOps telling how a project has to be run because DevOps is product centric (more details on the blog post and with search on this term).

"DevOps" is about breaking silos and have a team composed of developers, architects, ops, exploitation peoples who handle a software on its lifetime, that is antagonistic with the waterfall project approach.

To quote here Jez Humble:

  Products not projects

So your answer would be : There's not responsibilities for a project manager on a DevOps product because the main idea behind devops is to have a team handling the product on long term and not on a planned project time only.


Are there arguments against using “imagization" tools in the literature and if so, what are the main ones?

You might have better luck finding answers to your questions if you use the more standard term of "containerization".

This article discusses containerization as well as some pros and cons you might want to consider.


Mass remove/add to multiple security groups cross region

Do you already manage your resources with Terraform or are you looking for something to iterate through your AWS environment to change those settings?
If you're currently not managing your environment with Terraform, you'll have to import your resources into Terraform before you can begin making changes to your environment.
Your request sounds more like something you'd use the AWS CLI or perhaps Ansible for. Terraform isn't intended to perform scripting tasks on an environment.


How can I use a buildah script for image generation using GitHub Actions?

Here is what I did, I create a Makefile that includes my own method of image creation,
.PHONY: image clean image-release
image: image-release

image-release:
    cargo build --release
    buildah unshare ./image/distroless.sh release

clean:
    cargo -v clean

For this purpose the only thing that matters is the image created by ./image/distroless.sh matches the action (redhat-actions/push-to-registry) in the following flow. We'll continue using the name myAlpineImage as in the question,
name: Distroless Image Creation

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

env:
  CARGO_TERM_COLOR: always

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Download and create image.
      run: make image-release
    - name: Push To Registry
      uses: redhat-actions/push-to-registry@v2.1.1
      with:
        image: myAlpineImage
        tags: latest
        registry: ghcr.io/evancarroll/project
        username: evancarroll
        password: ${{ secrets.GHCR_TOKEN }}

You can see here we make the call to make image-release which calls the buildah script and makes the image rootless.
Then I added the secret for GHCR_TOKEN to my repo. The token is generated in the "Settings / Developer settings / Personal access" tokens on github..


How to Shutdown Docker Stack on Container Failure

If you don't want the containers to restart, you can set the condition: to none. See https://docs.docker.com/compose/compose-file/#restart_policy

On the swarm manager, you could use docker stack ps to check the status of the containers on a regular basis and shut down the stack if the database containers aren't running (using cron or whatever). For example:

docker stack ps --filter "desired-state=Running" your-stack |grep database || docker stack rm your-stack.

If containers are not able to restart, they shouldn't show up in the list once they fail, and once none are found, you can remove the stack.


Coupling docker registry and source control

I was able to get an answer over here forums.docker.com Credit goes to dmaze.

Set up some sort of automated build system (“continuous integration” in the current trendy term). Docker is sufficiently mainstream at this point that any of the cloud-based or locally-installed CI systems can do it.

In your Dockerfile, use a LABEL to record the source of the build. That probably includes the commit hash from distributed source control (git, Mercurial), the branch name if relevant, any release tags if present, and possibly details like the timestamp of the last commit. docker history and docker inspect should be able to show these.

When you docker push your images, push them at least twice, with the commit hash and with the branch name as the “version” part (quay.io/mycorp/imagename:123abc7, quay.io/mycorp/imagename:dmaze-test). If release tags are readily available, the CI system should push images with these tags too.

Make sure the Dockerfile is committed to source control, of course, and try to have a stable path to get any external dependencies that may be present.

Now you can go both ways: given an arbitrary commit, if your CI system built it, you can docker run the image it built; and if you have an image, you can find where exactly in source history it came from, and git checkout or hg up to that specific version, and docker build a near-identical copy of it yourself.


Jenkins Helm Chart - kubernetes-plugin pulling image from a private gcr

Found the issue, had:
kubectl -n jenkins patch serviceaccount jenkins -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}'

... applied on default serviceaccount instead of jenkins.


Kanban to catch a quick-moving situation

We are a product development shop - and in fact we build a Kanban tool (SwiftKanban), in addition to others.  We have a lot of experience with Kanban and DevOps and separate boards for epic/ story detailing vs. dev/ ps tracking. Our DevOps setup is still being fully automated and being gradually fleshed out. We've been on this journey since 2010-11 - and we have achieved significant business benefits exactly of the kind you are looking for.

We started with a single Kanban board, but with 2 separate swim lanes for the product manager and the dev team - see below -

Planning Lane for the Product Owner

Dev Lane for the main dev activity

Since we had a single team doing development as well as maintenance/ customer related work, as also refactoring work, we also had a few other lanes to help them visualize their overall workload -

Over the last 6 years, we have evolved, as have our dev/ engineering practices.  While we never used Scrum - and don't have sprints, we do production releases every 4-6 weeks, we deploy to an internal staging server (which we use as our internal production server, where we use our product for dev as well as other functions such as marketing, sales, support, implementation services, etc.) every week or less, and we have a single workflow between Dev and Ops as you will see below.

From that separate planning lane, we have evolved to a full upstream Kanban board - which is our roadmap planning board, where we track Themes, Epics and Stories - and lay out stories in up to 4 releases out. We did this as we gradually figured out just how much work needs to get done to identify and detail the work to be done as well as to get stakeholder alignment so everyone agrees that we are working on the right stuff.

Roadmap Planning Board

Stories from this board get pulled to the Dev board.  Internal and customer reported defects are directly added to this board's Ready queue.

Dev Board

As you can tell, the Product Owner (Product Manager) is responsible for the Roadmap board, while the Sr. Dev Manager is responsible for the Dev Board.  While Dev team members deply to Staging server, the Ops team ensures they have the right CI/ CD environment to do that.  After Dev/ Validation, it is the Ops Team's job to  deploy to production once sufficient valuable work has been completed (typically 4-6 weeks).

Both boards are accessible to all senior stakeholders as well as CEO, Sales and Marketing so everyone can track what is going on and what is prioritized for upcoming releases.

At the same time, we ensure we follow some of the recommended Kanban cadences, including the Daily Standup Meeting, the Weekly Replenishment Meeting (where fine-tuning of the story/ defect prioritization takes place) as well as Release Retrospective and a Quarterly Strategy Review Meeting.

The Kanban boards and cadences ensure that everyone is aware and aligned; and if there are conflicts, they are resolved during one of these meetings (other than the Daily Standup).

Besides key Kanban practices - including a reasonably strict adherence to WIP Limits - we have also, of course, invested heavily in engineering practices to enable a true continuous delivery/ deployment environment - and that journey still continues.  We are a Test Driven Dev shop and developers are responsible for JUnits, Functional Automation, and Peer Review.  Our code repository is Subversion and integration/ deployment is orchestrated using Jenkins. Subversion is integrated with our Kanban tool so all work at each stage is also updated on the card as well (via comments on the card).

Overall, to answer some of your questions more specifically,

Yes, it makes a lot of sense to provide a separate board to the product owner and another to the dev/ ops teams and link cards across them as needed.
And it makes sense to organize the board by tasks/ workflow stages rather than people as that gives you greater flexibility with using cross-functional folks in different stages of the workflow.
Most importantly, it lets you easily visualize and measure which workflow stages might be bottlenecks (rather than which people - which would typically be inaccurate) and what tweaks you might need to make to improve your flow and throughput and reduce lead time.

All the best with your transition to Kanban! If you need any more help with respect to Kanban, I strongly recommend reading David Anderson's book on the Kanban Method as well as refer to our comprehensive Kanban guide.

Hope this helps.


vagrant on windows 10 incorrect network configuration on multi-machine setup

eth0 is used by vagrant for nat configuration in order to establish portforwarding. eth1 private networks that you have configured should allow you to ping/nslookup from one node to another. You can use the following to customize the ip of the nat but I don't see why would you do that:

config.vm.provider "virtualbox" do |vbx|
  vbx.customize ['modifyvm', :id, '--natnet1', '192.168.169.0/24']
end

Update: tested your setup it doesn't work, this is some CentOS 8 specifics.


Why is ansible not sending the correct sudo password?

The cure (or bandaid) was to set Pipelining=False in the ansible.cfg file.

This kind of stinks because I like Pipelining and it works, but I guess ansible is gonna treat that first ssh statement as something 'it owns' even though it's a local action.


How to define variable in Jenkins declarative pipeline?

The variable must be defined in a script section.

pipeline {
    agent none
    stages {
        stage("first") {
            script {
                 foo = "bar"
            }
            sh "echo ${foo}"
        }
    }
}


What are best practices to implement Gatling in CI?

https://gatling.io/docs/2.3/extensions/jenkins_plugin/

https://wiki.jenkins.io/display/JENKINS/Gatling+Plugin

  Configuration

Install Gatling Plugin (via Manage Jenkins -> Manage Plugins)
Configure your project to execute Gatling simulations, for example using the Maven plugin (see Maven plugin documentation)
Configure your job :
    For a maven job: add "Track a Gatling load simulation" as a new post-build action. Beware that neither your Jenkin's path nor your

  job's name should contain any space.
          For a pipeline job: add a line to your pipeline script: gatlingArchive()

  Optional configuration: You can publish Gatling results with the
  Jenkins JUnit plugin. Your Gatling simulation needs to have some
  assertions, as each Junit testcase is a different Gatling assertion.

Configure the Jenkins plugin with the following Test report XMLs: target/gatling/assertions-*.xml
Execute Gatling with the following option: -Dgatling.useOldJenkinsJUnitSupport=true, for example: mvn gatling:execute -Dgatling.useOldJenkinsJUnitSupport=true


Does Prometheus expose the Horizontal Pod Autoscaler's "Current CPU Utilization" as shown in the Kubernetes dashboard?

As we can see from kube-state-metrics/docs/horizontalpodautoscaler-metrics.md

We just need to have kube-state-metrics along with HPA to start getting metrics in prometheus.


Jenkins Artifactory plugin not downloading in multi-thread

You could try to run the artifactory file upload in parallel if you are using the Jenkinsfile syntax: https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/parallel-from-list/parallelFromList.groovy

Here is a simpler example to run things in parallel if you only have a fixed number of things you want to do in parallel:

parallel (
    "task1" : {
        //runTask1
    },
    "task2" : {
        //runTask2
    },
    "task3" : {
        //runTask3
    },
)

This can be enclosed inside a node, in which case they would all share the same workspace or each command can define node inside the task code to run into another node in which case it would have a different workspace for each task.


Best learning resources for continuous integration tools like Jenkins

Here are the  resources which might be useful for . continuous integration
Below . you might be visited but as novice . it will be easy to understand.

jenkins:-

https://www.tutorialspoint.com/jenkins/jenkins_git_setup.htm

Shell Scripting :-
 I would . suggest . to start with basic unix commands  like . cp,mv, file related and . permissions etc.. (Basic stuff).

Its good to have . some practise  on the system so that you can  get an Idea . Because as a Devops engineer we should be strong enough on Basic . stuff.


How to update nodejs on Jenkins master node?

I would have a look at the Tool Auto-Installation which allows Jenkins to install the tooling you need.

  Lets you configure tools so that agents will install them on demand whenever running a job that needs them. This could be especially useful when running a large farm of agents in a cloud which all start with a minimal operating system configuration.

Speaking of that also checkout the NodeJS Plugin which

  Provides NodeJS auto-installer, allowing to create as many NodeJS installations "profiles" as you want.
  The auto-installer will automatically install a given version of NodeJS, on every jenkins slave where it will be needed
  Allows to install globally some npm packages inside each installations, these npm packages will be made available to the PATH


Ansible configure and compile in directory

Ansible command module requires only one command and cannot be changed. You should use the shell module at the very least. Note however that this will break the idempotency of the playbook.

You should probably split the task up into three:

configure task using the shell module. Use creates: to tell the task to be skipped if the configure task has already created the expected output file
make task, using the Ansible Make module
similarly for the make install task.

This should yield the same actual result, but would be more likely to be idempotent.


Thaks for all answers.
Teamcity began to work correctly when I installed all windows updates to my Windows Server 2012 R2.
Maybe it depended with last Microsoft security update.


How to probe all running aws ec2 instances and their sec groups, etc to generate templates?

Terraform would be the best resource to create these EC2 instances repeatedly.

I would highly recommend you to create a variables.tf or terraform.tfvars file template allowing you to create a new instance of these resources and store that file and the state file securely away for future management of the resources. A versioned S3 bucket would do quite well for this.

Please note that once you create your Terraform code you can instruct Terraform which state file(s) to work with, etc. Allowing you to reuse the same code over and over without duplication.


you create a Docker Swarm stack file:

---
version: '3.1'

services:
  ubuntu:
    image: ubuntu # or your custom Docker image
      deploy:
        replicas: 10000

Then, with docker stack you can run your 10000 Ubuntu's on one or - probably better in this case - a set of Swarm hosts. This could be bare metal or AWS. Enjoy!

For MAC address is very interesting question which I hope other colleagues here can answer. Geolocation should be no problem as this is AFAIK a system setting - virtual OS does not implement a physical GPS sensor.

Docs give an example how to set a fixed MAC address on docker run - possibly this can be translated to the stack YML file as well.

docker run --mac-address=".." ...


How to disable ping thread on jenkins agent

In the web UI, from the home page, navigate to Manage Jenkins > Manage Nodes > Configure.  This page has all of the health checks that Jenkins performs to determine if an executor is up.  The check you are concerned about is called "Response Time".  You can disable the "Response Time" health check on this page.

From the help text on the same page:

  This monitors the round trip network response time from the master to the agent, and if it goes above a threshold repeatedly, it marks the agent offline.

  This is useful for detecting unresponsive agents, or other network problems that clog the communication channel. More specifically, the master sends a no-op command to the agent, and checks the time it takes to get back the result of this no-op command.


Push specific changes (defect fix) and not feature changes

This sounds like a problem for git cherry-pick. If you can separate the user story code from the fix code, by committing them separately, you can cherry pick the relevant commit to merge back into the develop branch.

See https://stackoverflow.com/questions/9339429/what-does-cherry-picking-a-commit-with-git-mean


Safari test automation using Robot Framework via Jenkins

Do you run selenium as a separate process?
I had the same issue if you are not running selenium in the background.

(edit)
Try java -jar selenium-XX.XX.jar in terminal.

After that run pabot or robot command to run the tests.
You can crate app through the Automator and use the same approach you describe to call it from the CI. That is what I end up using.


What can Vagrant, Puppet, Docker, Chef do for a Full stack Web developer?

First of all, I strongly advise against "feature comparisons" blog post for similar software, they're quickly outdated and I'll try to keep this answer generic for a system configuration manager (SCM) and let docker out of the talk for the first part.

What it can brings you:

Reproduction of configuration from environment to environment
Versioning of the configuration
Enforcing the configuration

To get a real life example for a PHP application you'll need:

an operating system
a webserver
php libs
configuration of your app for external services (smtp, database IP, etc.)

What a SCM can provide is starting from a minimal base OS:

taking care of installing the webserver (apache/nginx)
installing PHP (either by package or source)
installing the necessary libs
Setting up your site config (vhost in apache)
Setting up your php.ini with any tweaks needed
deploying your app from a repository (from git to nexus, wide range of choice)
Writing your application configuration (here can be the smtp or database server for this environment)
Starting the webserver
Eventually running some smoke tests on your app.

Now if someone comes to connect to the machine and do a manual change to any of the configuration, the goal of a SCM is to keep the machine in its desired state, at next run, the change will be detected and the file changed will be put back to its desired state.

This has the advantage of enforcing any change to be put in the desired state description and not be forget between environments. Usually chef or puppet are set up to run on a periodic schedule, default for Chef is 30 minutes.

Now the big difference with docker is that you will do the same thing, installing you middleware, libs, forcing a templatized configuration, etc.
Once in the runtime phase, if there's a breach in your app and an attacker use it to tweak your config or modifying your webserver configuration it can stay unnoticed for a while if it doesn't break the app.

A SCM will revert the changes if you use it properly it will even remove unwanted sites in the sites-enabled directory of apache and thus will raise the complexity for an attacker.

Likewise if your webserer dies, the container will stop and unless you have a system to monitor it an relaunch it you're down. A SCM will notice the service is stopped and restart it at its next run, less immediate than runit but it will correct a rotate script which didn't properly restart the service (like the default apache logrotate conf on Ubuntu  which may time out and let your apache stopped)

A SCM may be interesting even to create your docker containers, you know you'll have the same behavior on a centos or ubuntu machine (most community code handle using the proper package names if you go this way) and so it will remove problem of edit to package names in a dockerfile.

The main idea behind a SCM is to manage your infrastructure as you manage your application, code it, version it, test it and be sure it will always behave the same.

If you rely on human procedure for configuration of the middleware/libs/app config you can be sure there will be a step forget during the deploy process once in a while, and spotting this problem can be really hard. With a SCM you prevent this omission.


How to check minimum code coverage in pull request changes?

I found a solution to my problem. There is a library diff-cover which does exactly what i need. Here is a link -> https://diff-cover.readthedocs.io/en/latest/README.html


What are HiveQL limitations compared to SQL?

The Wikipedia page for HiveQL provides a brief summary of the differences:

While based on SQL, HiveQL does not strictly follow the full SQL-92 standard. HiveQL offers extensions not in SQL, including multitable inserts and create table as select, but only offers basic support for indexes. Also, HiveQL lacks support for transactions and materialized views, and only limited subquery support. Support for insert, update, and delete with full ACID functionality was made available with release 0.14.
Internally, a compiler translates HiveQL statements into a directed acyclic graph of MapReduce, Tez, or Spark jobs, which are submitted to Hadoop for execution.

One of the biggest limitations is with transactions, as documented in their wiki - it appears that ACID semantics have been added recently, so the support is not quite as mature as it would be in a typical RDBMS like MySQL.
Hortonworks have also published a cheat sheet for converting SQL to HiveQL - it looks pretty useful for determining the differences between Hive and a normal RDBMS, although it does appear to be slightly outdated (it talks about v0.10 and v0.11, when the latest release as of March 2017 is 2.1.1).


Is DevSecOps indeed a new practice worth a professional term?

In my experience, the purpose of the "DevSecOps" (or whichever order you prefer) has mostly to do with including Security as a first-order concern of software delivery.  Security is commonly treated as a "bolt-on" (often meaning optional,) non-functional requirement.  DevSecOps attempts to describe a process where security concerns are folded into the delivery process at every step of the value stream.


Jenkins: mercurial scm: clean=false is not working as expected

Seems like --clean is hardcoded into the MercurialSCM Jenkins plugin.

See OP for the link to the culprit code.

I have opened up a ticket on Jenkins jira.


How to ensure that removed allowed ports will be removed using Ansible's UFW module?

Based on:

# Allow OpenSSH. (Note that as ufw manages its own state, simply removing
# a rule=allow task can leave those ports exposed. Either use delete=yes
# or a separate state=reset task)

The following snippet was added to the top of the file:

---
- name: Reset UFW
  ufw:
    state: reset

Once ansible was run:

TASK [rolename : Reset UFW]

the undefined ports were removed


Can/should I use Jenkins to monitor whether a website is up and send out an email if it's down?

It's possible and pretty simple to do this with Jenkins. Here is a sample pipeline:

pipeline {
    agent any

    triggers {
        cron('*/5 * * * *')
    }

    stages {
        stage('Health checker') {
            steps {
                sh 'curl https://mywebsite.com'
            }

            post {
                failure {
                    mail to: 'notify-list@example.com',
                        from: 'jenkins@example.com',
                        subject: "${env.JOB_NAME} - Failed",
                        body: "Job Failed"
                }
            }
        }
    }
}


Anything that "needs" to happen in a given time frame is not a candidate for spot instances, including spot blocks.

Spot capacity is, by definition, surplus/idle capacity, subject to availability, so your spot block may simply fail to start, as capacity may not be available.  It is less expensive for exactly that reason -- AWS is selling surplus capacity for which no one is currently willing to pay more.

Information about the pool of compute capacity that is currently flagged internally as eligible for participation in the spot market is not publicly accessible, and although it was possible to make some reasonable assumptions in the past about capacity based on signals you could obtain from the real-time pricing, the spot pricing algorithm was modified in 2017 in a way that largely masks those signals (and the change, in my opinion, was a good thing for any customer who was not aggressively gaming the system).

Spot blocks are intended to run for a committed period of time, so they are less likely to be interrupted than other spot instances, but they can initially start up successfully and then subsequently be interrupted at any time.

  In rare situations, Spot blocks may be interrupted due to Amazon EC2 capacity needs. In these cases, we provide a two-minute warning before we terminate an instance, and you are not charged for the terminated instances even if you used them.

  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#fixed-duration-spot-instances


How to avoid OOMKilled with azcopy on k8s

azcopy (v10) tries to use all available memory to buffer the blobs on transfer. It seems that the calculated memory size is incorrect and exceeds the pod memory limit.

This can be controlled with the option AZCOPY_BUFFER_GB, as described in the azcopy env output below.

  INFO: Name: AZCOPY_BUFFER_GB

  Description: Max number of GB that AzCopy should use for buffering data between network and disk. May include decimal point, e.g. 0.5.
  The default is based on machine size.

So for a pod limit of 2GiB, you can set the buffer a bit lower:

export AZCOPY_BUFFER_GB=1.5
azcopy bench ...


How to add job dependency to gerrit trigger in Jenkins pipeline?

The "correct" way to determine the names for Pipeline step parameters is to use the Pipeline Snippet generator in the web UI:

https://www.jenkins.io/doc/book/pipeline/getting-started/#snippet-generator

However, based on browsing the source code, I am guessing the name of the parameter you are looking for is dependencyJobsNames.


Internal network between container Docker Compose with outgoing connection

Internal Docker networks mean there is no gateway configured to reach the outside internet, you can only reach other containers from that network. In your scenario, none of your networks should have this definition.

Without setting the internal network flag, containers can reach out of the docker host, potentially to the internet if the host has that access. This is one-way access, containers can access the external network, but that external network cannot access the containers.

To access a container from outside of the Docker host, you need to publish the port on the host mapping into the desired container (or service). Publishing ports with the ports section in the docker-compose.yml file is only needed to provide external access, not access between containers. Applications inside containers need to listen on all interfaces (0.0.0.0) for both publishing ports on the host and for container to container communication. However, you can publish ports on a specific interface on the host, e.g.:

ports:
  - '127.0.0.1:8825:8825'
  - '127.0.0.1:8835:8835'

For container to container communication, it is not necessary to expose or publish the port. All that is needed is a user defined network (which you have) in common between the containers. Then the containers can communicate using Docker's built-in DNS to resolve the service name (e.g. consul and mongo) and the application port inside that container (e.g. consul:8300). To avoid the external network from accessing these containers, do not publish any ports from them.

Side note: from your docker-compose.yml, you should also consider removing container_name, depends_on, and build since they won't work if you try to transition to swarm mode.


I don't know of a built-in way but here are two hacks:

You can look at https://dev.azure.com/{organization}/_settings/audit , then set the date range to what you want and manually count.

An automated way of doing that could be to use the ADO REST API and send a Query to get the Audit log, then use a bit of code to tally up the unique users from the response.


Azure - Connection String in Key Vault vs Application Settings

The benefits as I see them are the general reasons to use Azure Key Vault

Secrets are centrally stored with options for authorization, auditing, etc.
Azure can be scripted to update the secrets on a timer - so in case a connection string gets in to the wrong hands, it is only valid for a day (or a week, or whatever)
In case the secret is shared between multiple applications, you only have to update it in one place (not the greatest benefit, but still nice)


withEnv doesn't persist environment variables.  Instead, you need to tell withEnv what steps to run with that environment by passing it a block.   Here is what I think you want your code to look like:

#!groovy

node('superhost01'){
  String HOSTNAME="host01"
  String USERNAME="tech_user"
  withEnv(['PATH=/data/jdbc_connector']) {
    stage('Prepare') {
      checkout scm
    }

    stage('Deploy') {
      sh """
      scp -r config.yaml ${USERNAME}@${HOSTNAME}:$PATH
      """
    }
  }
}

As a side note, I suspect you actually want to set PATH to /data/jdbc_connector:$PATH, otherwise your checkout step will not be able to find git and your deploy step will not be able to find scp.


Google Container Registry giving 'Access Denied' when trying to push Docker container

In case anyone else is running into this issue and can't figure it out, I fixed it by running this command:

gcloud auth activate-service-account --key-file key-file.json

And then ran my docker push command and it pushed through.


Sure, you can do this with Scripted Pipelines.  I've never tried to do it with Declarative so I'm not sure whether it's possible to do with Declarative or, if it is, if it's easy or straightforward.

Here's an example of what this might look like with a Scripted Pipeline.  I haven't tested it so there may be syntax errors or other problems.

def parallelTopLevelSteps = [:]

parallelTopLevelSteps['A and B'] = {
  def parallelNestedSteps = [:]

  parallelNestedSteps['step A'] = { echo('A') }
  parallelNestedSteps['step B'] = { echo('B') }

  parallel(parallelNestedSteps)
}

parallelTopLevelSteps['C and D'] = {
  def parallelNestedSteps = [:]

  parallelNestedSteps['step C'] = { echo('C') }
  parallelNestedSteps['step D'] = { echo('D') }

  parallel(parallelNestedSteps)
}

parallel(parallelTopLevelSteps)


How to Save and Restore the Iptables rule and configuration from file?

If you want to modify the configuration and rules, so you must save the current configuration to a file.
So step one is to save the rules configuration by typing the following commands:
$ sudo iptables-save > /root/my-iptables.rules

To restore it just use the command iptables-restore:
$ sudo iptables-restore < /root/my-iptables.rules

⚠ When you changed the existing rules after saving the file, You should save it again or change the existing file and restore it from the file.


How to list the containers attached to a given user-defined network?

The docker network inspect command produces a JSON output that can be parsed using jq to extract the relevant piece of information:

sudo docker network inspect app-bridge | jq '.[].Containers[].Name'

"app2"
"app3"
"app1"
"laughing_easley"
"database"
"thirsty_wing"

If you are already familiar with JSON processing with jq, the advantage is you can reinvest your knowledge without having to bother with Go templates. On the other hand, it relies on an external tool which may or may be not available at your site.


Is it possible to read secured keys from aws-secrets-manager without using aws access and secret key?

create a service which authorizes against SSO and the provides temporary credentials via AWS STS Service. That way you don't have to store anything anywhere and once the user loses access to SSO they will also lose access to AWS. You need a role with the access to your dev environment and the developers will assume that role.


How to run Ansible with sudo no password commands with become syntax

When I run ansible against a host machine that has some commands I need to run as root, I can't run these with "become" because it runs the commands as root instead of just running them with sudo.

That's not true; become uses a variety of methods for privilege escalation, and defaults to using sudo.
Are you perhaps running into one of the known limitations with the become module, like trying to limit sudo access to certain commands?


How to connect to external etcd server inside of docker container (python application)?

The error you see in your container is:

  etcd.EtcdConnectionFailed: Connection to etcd failed due to MaxRetryError(u"HTTPConnectionPool(host=u'localhost', port=2379): Max retries exceeded with url: /v2/keys/my_module/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))",)

If you take a close look there, you can see that it's trying to connect to localhost.  Inside your container, localhost is means "the current container" (much like localhost on your host means "the current host"). If you want to connect to an etcd instance outside of the container, you'll need to use the ip address (or hostname) of another server.

If you're running (a) running Docker locally, (b) etcd on your host is listening on all addresses, and (c) there are no firewall rules that would otherwise prohibit the connection, then you can use the address of the Docker bridge associated with your container as the address of your host.  This will be the default gateway as seen from inside the container.  You can extract the address with something like:

ip route show | awk '$1 == "default" {print $3}'


Is there any relationship between "production engineering" and DevOps?

According to Facebook

  Production Engineering at Facebook is a hybrid between software and
  systems engineering; it keeps Facebook running smoothly and scaling
  efficiently. The Production Engineering team works within Facebook’s
  product and infrastructure teams to make sure our services are
  reliable and scalable. The team writes code and debugs hard problems
  in live production that impact more than 2 billion people around the
  world. From backend services like our Hadoop data warehouses, to
  frontend services like News Feed, to infrastructure components like
  our caching infrastructure, load balancing and deployment systems, the
  Production Engineering team keeps Facebook running.

as team works within product and infrastructure meaning it works along with developers and operations.


What are you trying to achieve?
If your goal is to run build agents to execute pipelines on premises, then you don't have to allow inbound access. Build agents will query Azure DevOps for any new jobs, so the connections are actually outbound.


How to locally resolve host name starting with variable slug?

People often use dnsmasq for this.  You can wildcard sub-domains like this fairly easily for local testing using it.

Here's an example - https://qiita.com/bmj0114/items/9c24d863bcab1a634503 but there are plenty online.


Is it possible to install bitbucket container on Centos?

Since I do not currently have access to a server with Centos installed, I tested my soultion on Ubuntu 20.04. That being said, the whole point of using Docker is to make applications distribution-agnostic, so it should work on Centos as well.
You're getting an image does not exist error, because the image you specified does not, in fact, exist. There is a typo in one line of the README file provided on Docker Hub and you happened to use that line when writing your Compose file.
That being said, incorect image name is not your only issue. You have not specified a volume for your Bitbucket installation, meaning all of your data will be lost if the container is destroyed. Furthermore, the only way to backup a dockerized Bitbucket installation is to use method described here, which requires using external database.
With that in mind the minimal Compose file for Bitbucket is:
version: '3.3'
services:
  bitbucket:
    hostname: bitbucket
    image: atlassian/bitbucket-server:latest
    environment:
     - JDBC_DRIVER=org.postgresql.Driver
     - JDBC_USER=bitbucketuser
     - JDBC_PASSWORD=jellyfish
     - JDBC_URL=jdbc:postgresql://db:5432/bitbucket
    volumes:
      - bitbucket_data:/var/atlassian/application-data/bitbucket
    ports:
      - 7990:7990
      - 7999:7999
    depends_on:
      - db
  db:
    hostname: db
    image: postgres:11.2-alpine
    command: -c 'max_prepared_transactions=64'
    environment:
     - POSTGRES_USER=bitbucketuser
     - POSTGRES_PASSWORD=jellyfish
     - POSTGRES_DB=bitbucket
    volumes:
     - db_data:/var/lib/postgresql/data
volumes:
  bitbucket_data:
  db_data:

That can be further extended with httpd or nginx service used as a reverse proxy, which could also be used to enable HTTPS connections to your Bitbucket instance and an Elasticsearch service to replace the embedded one as well as an email server for Bitbucket to use to send various notifications to users.
I would definitely recommend the SSL-enabled reverse proxy and email server for production. Elasticsearch server is only necessary if you want to run Bitbucket cluster rather than a single instance.


Is a decentralized Docker swarm possible?

By Docker Swarm design it is dencentralized, so you can promote any node to a manager. Normally, 3 managers are enough, for landscapes with thousands of nodes 5 are enough.

https://docs.docker.com/engine/swarm/admin_guide/


Getting a 502 Bad Server Request on Nginx

Never mind, I just figured it out . I just needed to change the port i was refering inside the backend file /etc/nginx/sites-enabled/backend
Changed this line proxy_pass http://localhost:9000; to this line proxy_pass http://localhost:3000; and it worked.


Definition of change failure rate and mean time to recovery DORA metrics

You are asking the right questions and hitting on a very important aspect of these metrics. Individual metrics are not super helpful and measurements need to be defined. Having worked with these measurements across small and large organizations, let me walk you through my thought process.
Definitions:

Metrics - Some sort of calculation on one or more measures
Measure - An amount of something using a specific unit
Change Failure Rate (CFR) - The percentage of deployments causing a failure in production
Mean Time to Recover (MTTR) - How long it takes an organization to recover from a failure in production

This conversation could include Service-Level Indicators (SLI), Service-Level Objective (SLO), and Service-Level Agreement (SLA), however, for this purpose I will assume we have measurements (SLI) to ensure we are meeting our goals (SLO) and not breaking our promise to users (SLA).
Before getting into the details, the main question asked is:
What are we trying to measure?
At the end of the day, any organization exists to create value, these metrics are meant to track any interruption in the flow of value from the organization to its consumers / users.
As an example, if we wanted to measure how efficiently the organization adds new value to users,  Lead Time for changes is a great way. How long does it take new code or business ideas (depending on if you would like to include non technical business processes) to get from creation / ideation to the user? Lead Time includes CFR and MTTR, both of these can interrupt getting value to the consumer and increase Lead Time.
Using the value creation lens, let's look at CFR first.
What does the organization consider a failure in production?
I have typically used it as an event needing immediate fixing, on the call during the production deployment. The service we deployed needed to be rolled back, a hotfix needed to be pushed immediately to correct an issue discovered during the deployment, or something equally catastrophic. These are all interruptions to value reaching the user. A bug not presenting itself for a few days, would not create a Change Failure with this definition. However, if the bug began to trip indicators (SLI) (example, degraded speed or experience for the end user), exceeds our objectives (SLO), or even breaks or promise to users (SLA) then an incident is created and it will bring us to your next measurement, MTTR.
Assuming the bug mentioned creates an incident, MTTR measures how long it takes to fix it. If the fix is a roll back for the original release, or a hotfix of the release, it really does not matter. The measurement is how long it takes to correct.
Pulling this together.

Determine what the measure is.

For CFR, what does the organization consider a production failure? I provided a few examples, but all organizations are free to create their own measure. For MTTR, the measure is time, hopefully from issue creation to correction, sometimes we need to settle for incident discovery to correction.

With the Measure, track the Metrics.

How many failed production releases are there? How long, on average, does it take to correct production incidents?

With the Metrics, track and, hopefully, improve the flow of value from the organization to the consumer.

If a deployment is successful, it should not trip the measurement for Change Failure, however, if there are bugs, their interruption to value will be caught with Mean Time To Recovery.
These measurements and metrics (Lead Time To Change, Mean Time to Recovery, Deployment Frequency, and Change Failure Rate) are meant to tell a story in aggregate.
Assuming weekly deployment, if an organization has 0% CFR, but a 14 Month MTTR with incidents occurring every week, then the flow of value is constantly interrupted and the amazing CFR score means very little. Or, if the CFR is 50% and the MTTR is 2 Mins, then the high CFR looks horrible, but the actual interruption to users is only 2 Mins * bugs per week. It really takes both, or all four of the metrics, to create an accurate picture of the real state of the code and deployment process.
Hopefully this helps. Working through these measurements and metrics is a journey, probably with no conclusion, with a goal of improving the amount of value an organization can create for its consumers.


Terraform Issue - Can't find Data Source in Sub Module?

You need to pass that info to your module or also declare it in your module again. For example, for the region, pass it as a variable to the module:

module "foo" {
  ...
  current_region = "${data.aws_region.current.name}"
  ...
}

Or add it inline in your module, just like you did in your main.tf


Ansible YAML inventory - Groups that are allways members of other groups

According to this best practices post, your file will look like this

file: test

[webservers]
web1
web2
web3

[mysqlservers]
db1
db2
db3

[lamp]
lamp1
lamp2
lamp3

[misc]
host1
host2

[all:children]
misc
webservers
mysqlservers
lamp

[webservers:children]
lamp

[mysqlservers:children]
lamp

You can analyse your file by using ansible-inventory:

ansible-inventory --inventory-file=./test --list

output:

{
    "_meta": {
        "hostvars": {
            "db1": {},
            "db2": {},
            "db3": {},
            "host1": {},
            "host2": {},
            "lamp1": {},
            "lamp2": {},
            "lamp3": {},
            "web1": {},
            "web2": {},
            "web3": {}
        }
    },
    "all": {
        "children": [
            "lamp",
            "misc",
            "mysqlservers",
            "ungrouped",
            "webservers"
        ]
    },
    "lamp": {
        "hosts": [
            "lamp1",
            "lamp2",
            "lamp3"
        ]
    },
    "misc": {
        "hosts": [
            "host1",
            "host2"
        ]
    },
    "mysqlservers": {
        "children": [
            "lamp"
        ],
        "hosts": [
            "db1",
            "db2",
            "db3"
        ]
    },
    "ungrouped": {},
    "webservers": {
        "children": [
            "lamp"
        ],
        "hosts": [
            "web1",
            "web2",
            "web3"
        ]
    }
}


Best practise/standard practise for semantic versioning in development

Version bumps should happen every single build.  Waiting until a formal release and releasing everything as 1.0.0 is an option, but you should still know what build you are using in your alpha/beta.

Ultimately, the developer will have to determine if a major release increment is needed.
You should only do a major release bump if you introduced a breaking change.

For your regular builds, a common practice is to let your CI tool increment the "patch release" number or the X.X.Patch portion of the version.  This way your versioning is tied to your builds which will help the development and testing process.  If you choose to only do a nightly build, you can have your CI tool increment this number then so that you and other developers know what version of the package they are using.

NPM has published a good guide on how to handle semantic versioning.


What are best practices for running Pipeline tests into a CI framework?

Unit testing in an alternate CD/CI platform would be overkill in every situation I've built one.  The shared-pipeline code repo you connect to Jenkins will be triggered by a Jenkinsfile.  I suggest you define any and all testing in that particular Jenkinsfile - and keep this one simple enough that extensive testing of the Jenkinsfile defining tests for your Jenkinsfile does not also need a Jenkinsfile defining tests.


Create and add token to be able to connect to SonarQube.
You have create project in SonarQube and use it as a parameter:

sh """
   ${scannerHome}/bin/sonar-scanner \
   -Dsonar.projectKey=your_project_key_created_in_sonarqube_as_project \
   -Dsonar.sources=. \
"""

Copied from [StackOverflow] Problem with Connecting Jenkins to SonarQube.


Jenkins Pipeline Docker Agents with official Jenkins Docker image

You have to extend the jenkins/jenkins:lts to install the Docker client:

FROM jenkins/jenkins:lts

# Switch to root
USER root

ENV DOCKER_VERSION      docker-18.06.3-ce
ARG DOCKER_GID=993 # put the correct docker gid

# Download and install docker client
RUN wget --quiet -O- \
  https://download.docker.com/linux/static/stable/x86_64/${DOCKER_VERSION}.tgz | \
  tar zx --strip-components=1 -C /usr/local/bin docker/docker \
  && groupadd -g ${DOCKER_GID} docker \
  && usermod -aG docker jenkins

# Switch back to jenkins user
USER jenkins

You can now declare pipelines with docker agents.


What tool or type of tool am I looking for?

Jenkins can work just fine for this purpose.  In my view, Jenkins is a general-purpose automation tool, not necessarily a CI/CD tool specifically.
I'm on an Ops team and we use both Jenkins and Rundeck.  We generally prefer Jenkins because we find it difficult to navigate the Rundeck UI and find things like job output, but on the other hand it's often easier to write jobs for Rundeck, because you don't need to know Groovy/Pipeline.  Here are some examples of what we use each for:

Install package updates on managed systems (yum/dnf packages) - Rundeck
Deploy & provision new VMs - Jenkins
Run Chef on managed systems - Rundeck
Clean up stale Computer objects in Active Directory - Rundeck
Clean up stale subscriptions in Red Hat Satellite - Rundeck
Create & configure new repositories in Bitbucket from templates - Jenkins
Build new vagrant images - Jenkins
Backup & upgrade some services that my team manages (AWX, Red Hat Satellite, etc.) - Jenkins or Rundeck depending on the service

I'm not claiming that either of these tools is the best choice for any of these particular examples, just that my team uses them for these purposes and they generally work well.  Historically, which tool we use for each job generally has come down to personal preference of the person who created the job.
In terms of the name of this type of tool, we just call them "automation tools" or "automation platforms".  I'm not sure if there's a more "industry-standard" term for this type of tool.


Your question is not silly at all, indeed managing WordPress at scale requires broad knowledges due to how WordPress is implemented.
It's good to know that WordPress consists of 1) core PHP code, 2) plugins/themes and 3) uploaded media assets.
To serve the core and plugins, you'll need to locate the code either in local disk or NFS. Local disk is obviously faster since it doesn't interact with network, but NFS is a good choice if you want to have multiple servers to share the same code.
The uploaded media assets are a bit different. The most common "assets" are image files, and you can completely offload the server traffic to object storage such as AWS S3 or GCP Cloud Storage. Unlike plugins and themes, the media assets  are static meaning it will never run server-side logics like PHP.
Moreover, WordPress is read-heavy application that most of the MySQL query is to render blog posts to your public audiences, means any-sized server may blow up when unexpected number of visitors rush to your website. This is where CDN shines, to cache rendered HTMLs in the edge servers at the first visit, then serve the cached files without reaching to origins. AWS CloudFront and GCP Cloud CDN are popular CDNs to turn cheap WordPress server to globally-scalable media site.
So, I recommend to have the following architecture:
Visitor --> CDN --> Web/PHP Server --> MySQL

When configured correctly, you can serve millions of traffics backed with $10 web server.


declaratively mount a service account secret in OpenShift

Okay the answer to this is that you can set spec.serviceAccountName on the dc and the secrets are mounted.

Annoyingly the same does not hold true for BuildConfig objects. They have an inconsistently named spec.serviceAccount. Setting that doesn’t mount any secrets associated with the sa. It also means your build isn’t an “image puller” until you explicitly set that up. So this configuration option appears pretty useless as at openshift 3.11.


Reconciling Continuous Deployment with strict change management policies

If you must adhere to the change process, you'll be limited according to the limitations of the change process, full stop. If changes must be approved prior to deployment, you cannot do continuous deployment. If approval takes a long time, you cannot deploy quickly. There's no workaround whereby you can both follow the process and not be impacted by it. That's the cost of following the change process, and it's a cost worth bringing to the attention of the stakeholders in that process.

All is not lost... you can maximize the automation around the process, in order to minimize errors; all of the CD steps except for the linkage between generating a stable artifact and deploying that artifact to production. That linkage would be replaced by either some kind of user intervention (button, CLI command, etc.), or linked to the record of approval (e.g. when a change request ticket is moved to the "approved" state, fire off the associated deployment). You just have to squeeze as much benefit as you can out of it, while following whatever mandatory process you've been saddled with. It won't make approvals go any faster, of course.


How to connect a build node to a Jenkins master?

Connecting a node to Jenkins through a terminal only is a two step process.

First, you should ssh into into your node machines and set them up with the following steps:

Make a jenkins user shell:  sudo adduser jenkins --shell /bin/bash
Create the directory after switching to the jenkins user:  mkdir /home/jenkins/jenkins_slave
Create your SSH credentials to be used in the following bash script.

Then, once your nodes are set up, you will need to connect to your jenkins machine and run bash script such as this one to connect your nodes to your master:

#!/bin/bash

JENKINS_URL=$1
NODE_NAME=$2
NODE_SLAVE_HOME='/home/build/slave'
EXECUTORS=1
SSH_PORT=22
CRED_ID=$3
LABELS=build
USERID=${USER}

cat <<EOF | java -jar ~/bin/jenkins-cli.jar -s $1 create-node $2
<slave>
  <name>${NODE_NAME}</name>
  <description></description>
  <remoteFS>${NODE_SLAVE_HOME}</remoteFS>
  <numExecutors>${EXECUTORS}</numExecutors>
  <mode>NORMAL</mode>
  <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
  <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.5">
    <host>${NODE_NAME}</host>
    <port>${SSH_PORT}</port>
    <credentialsId>${CRED_ID}</credentialsId>
  </launcher>
  <label>${LABELS}</label>
  <nodeProperties/>
  <userId>${USERID}</userId>
</slave>
EOF


Docker Build Ending Up `returned a non-zero code: 1`

Line 81 of build.sh is trying to run python3.  But the problem is more fundamental because you aren't getting any of the earlier echos or apt-get command's output.  I would try:
chmod +x build.sh
./build.sh

to let the sh-bang line do its job.


Prometheus-operator - detection of expiring certs failing

The alert is broken, it is fixed in further release version of the mixins, we need to apply on(job) infront of the histogram to fix it. Example: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) <  60432131800 and yes, that means your certs are not being tracked...


I'd recommend a temporary container for this. It will work on just about any version of docker, and supports named volumes with non-default settings, e.g. that store data on remote systems (e.g. NFS) or modify the bind mound settings to use another folder.

A simple example is:

$ docker container run --rm -v "${volume_name}:/volume" -w /volume busybox df -P .
Filesystem           1024-blocks    Used Available Capacity Mounted on
/dev/..........              898086280 350169736 502226484  41% /volume

You can parse that output or reformat with the df command as appropriate for your use case. The busybox version of df will have fewer options than you may be used to:

$ df --help
BusyBox v1.31.1 (2019-12-23 19:20:27 UTC) multi-call binary.

Usage: df [-PkmhTai] [-B SIZE] [FILESYSTEM]...

Print filesystem usage statistics

        -P      POSIX output format
        -k      1024-byte blocks (default)
        -m      1M-byte blocks
        -h      Human readable (e.g. 1K 243M 2G)
        -T      Print filesystem type
        -a      Show all filesystems
        -i      Inodes
        -B SIZE Blocksize


how to update .env file in production while Jenkins deployment

You can store secret file at Jenkins and use it for deployment.
It can be achieved by using Credentials and Credentials Binding plugins.

Install Credentials Binding and Credentials plugins
Select your user name in the top navigation bar.
Select Credentials in the left pane.
Under Stores from parent, select (global) in the Domains column.
Select Add Credentials in the left pane.

insert into your pipeline script

withCredentials([file(credentialsId: 'env', variable: 'mySecretEnvFile')]){
    sh 'cp $mySecretEnvFile $WORKSPACE'
}


Better method, thanks to the others responses for helping me get to this.

Determine what plugins are installed currently:

# certbot-auto plugins
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT

* nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator

* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Determine where your certbot (in my case certbot-auto) is installed:

# find / -name certbot
/opt/eff.org/certbot
...

Get into the Virtual Env and install plugin

cd /opt/eff.org/certbot/venv
source bin/activate
pip install certbot-dns-google
deactivate

Verify certbot plugins again

# certbot-auto plugins
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT

* dns-google
Description: Obtain certificates using a DNS TXT record (if you are using Google
Cloud DNS for DNS).
Interfaces: IAuthenticator, IPlugin
Entry point: dns-google = certbot_dns_google.dns_google:Authenticator

* nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator

* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


I think Ansible serial works only for the entire play, have you tried to use roles ?

For example:

- hosts:
    - web
    - db
  serial: 1
  roles:
    - { role: nginx, when: ansible_os_family == 'Debian' }
    - role2
    - role3


Found the problem:

I was using the human readable key name from the GUI.
Need to use the key's UUId ID instead (this is specified next to the name in credentials screen)

steps {
            sshagent ( ['THIS-SHOULD-HAVE-BEEN-A-UNIQUE-ID-INSTEAD-OF-A-NAME']) {


Cloud-init files are essentially bootstrap codes, that run before each startup, and can - among others - modify files, set up services, create users, etc.

Not all types of droplets support all functionalities of cloud-init, for example CoreOS uses it's own implementation, with a very limited subset of valid values.

To use this in terraform, simply provide the cloud-init file during droplet creation:

main.tf:

resource "digitalocean_droplet" "web" {
  image              = "coreos-stable"
  name               = "web"
  region             = "lon1"
  size               = "2gb"
  private_networking = true
  ssh_keys           = ["${digitalocean_ssh_key.dodemo.id}"]
  user_data          = "${file("web.conf")}"
}

web.conf:

#cloud-config
coreos:
  units:
    - name: "etcd2.service"
      command: "start"
    - name: "fleet.service"
      command: "start"

This will for example create a droplet, where CoreOS will run etcd2 and fleet during startup

You can find some more examples in this repository, where I show how one can use these configuration options to set up some simple docker based services on CoreOS


How can I push multiple containers, created with docker-compose, to a registry

Just make sure the images you build all have the right names in terms of registry prefix and so on. If not you need to rename them accordingly.

Otherwise this task is pretty easy. To push all Images you build using a docker-compose file just use docker-compose push.

Make sure you are authorised to push to the registry (logged in etc.)


CodeBuild - multiple input artifacts in AWS Pipeline

We've had exactly the same problem - our build stage needs to use information from two separate locations.

To solve this in a generic manner, I've created two general purpose Lambdas - one that will merge artifacts, and one that will extract specific files from an artifact. In both cases a new artifact is created that can be passed on to a subsequent CodePipeline stage.

See https://github.com/tooltwist/codepipeline-artifact-munge

If it does not meet your exact requirement, it should be easy enough to tweak to do what you need.


docker container for running automation scripts using jenkins

Yes, you are doing the correct approach. Or rather, "a" correct approach, in your specific circumstance.

You are proposing to build the image beforehand, not as part of each individual Jenkins job. Then your Jenkins job will consist of running the image; and in that ephemeral state (i.e., inside the container) you are checking out the test script, and running it.

The more usual alternative would be to leave the actual git checkout to Jenkins, and pull the script into your container with a bind volume. Jenkins has to be able to access your git repository anyways, so you get this checkout for free. There are two sides to the coin here: if you do the git checkout from inside the container, then you can run your test outside of Jenkins with no change whatsoever - say, during development, on your own machine. In any case, if you do the git checkout from inside, and if you need credentials to access the git repository, be sure to handle those securely. Don't put cleartext passwords in the image, or ssh keys with an empty passphrases.

The benefit you are getting from this is that your test environment (libraries, 3rd party test drivers and so forth) is well defined, so you can run it on any Jenkins node you have available. By not creating new images whenever your code changes, you avoid creating and distributing new images for every job.

  I do not need any product code, my scripts is all that is required

So you would never actually deploy those images anyways - hence it is perfectly fine to not build them each time. You still should be automatically building your images - no matter if you do it as part of the actual job, or as a separate job which only does it when the Dockerfile (and whichever files it pulls in with ADD / COPY) changes. Not for technical reasons, but to avoid a manual step.


Retrieve files deleted after Docker service rm

There's no special feature to undelete files on the host provided by docker. So if you delete your files using docker, you are left with the same option as when you delete on Linux without using docker:

Learn from your mistake and don't do it again (I'd recommend forwarding logs to an external log aggregator).
Restore from backup.
Do a forensic recovery of the hard drive contents.


How do I provision a Windows Docker container with Ansible?

I've always had issues with using ansible to provision Windows machines with Packer due to the winrm connections breaking, example issue.

A workaround which you can do is to use the shell-local on your host machine which calls the Ansible playbook with the new host (although you would need to give it the IP address of your Windows Docker machine).

I've added the example which I did to get a test working. This uploads ip.cmd to the Windows virtual machine, which gets its IP address, and is then downloaded back to the destination. The ansible playbook is then called against that.

"provisioners": [
        {
            "type": "file",
            "source": "ip.cmd",
            "destination": "C:/tmp/ip.cmd"
        },
        {
            "type": "windows-shell",
            "inline": [
                "echo [{{ user `ansible_group` }}] > C:/tmp/hosts",
                "C:/tmp/ip.cmd"
            ]
        },
        {
            "type": "file",
            "direction": "download",
            "source": "C:/tmp/hosts",
            "destination": "./ansible/hosts"
        },
        {
            "type": "shell-local",
            "command": "ANSIBLE_CONFIG=./ansible.cfg ansible-playbook -v -i ./ansible/hosts -l \"{{ user `ansible_group` }}\" -e \"ansible_user={{ user `username` }} ansible_password={{ user `password` }} ansible_become_pass={{ user `password` }} ansible_port=5986 ansible_connection=winrm ansible_winrm_server_cert_validation=ignore \" ./ansible/site.yml"
        }

where ip.cmd gets the IP address of the Windows host:

@echo off
FOR /F "tokens=2,3" %%A IN ('ping %computername% -n 1 -4') DO IF "from"== "%%A" set "IP=%%~B"
echo %IP:~0,-1% >> C:/tmp/hosts

source


Aggregating JUnit test results from multiple projects

The JUNIT plugin has a built-in publisher for aggregating test results called "Aggregate Downstream Test Results". You need to make sure that the 'parent' and 'child' jobs are linked together with a downstream/upstream relation. This is usually done by archiving an artifact in the parent build and copying it to the children builds using the 'Archive Artifact' publisher and the 'Copy Artifact' buildstep respectively.


Logstash multiple pipelines going into same index

There are two ways to accomplish this, though one of them was only available recently.

The old-school version, the one you can do as far back as Logstash 1.5, is to pay attention to tags and use conditionals to separate your inputs. Roughly...

input {
  tcp {
    port => 1525
    codec => json_lines
    tags => [ 'tcp' ]
  }
}

input {
  file {
    path => '/var/log/app.log'
    codec => 'json'
    tags => [ 'file' ]
  }
}

output {
  if 'file' in [tags] {
    elasticsearch {
      host => 'logstash-es'
      index => 'files'
    }
  }
  if 'tcp' in [tags] {
    elasticsearch {
      host => 'logstash-es'
      index => 'tcp'
    }
  }
}

This results in two inputs that output to two separate outputs. This is all one file, though. Elastic figured out people were muxing pipelines this way, and came up with a way to do multiple pipelines in separate files.

- pipeline.id: tcp-inputs
  path.config: '/etc/logstash/pipelines/tcp.cfg'
  pipeline.workers: 3
- pipeline.id: file-inputs
  path.config: '/etc/logstash/pipelines/files.cfg'
  pipeline.workers: 2

This approach is somewhat more maintainable since the pipelines are in separate files, and humans don't have to reason out how the flows work when presented in a single big file. Pipelines are available in Logstash 6.0 and newer.


How does Hashicorp's Vault compare to Lastpass?

I don't have enough rep to comment but I'm going to expand on what RuBiCK said.

In the first case, if you have a password for JIRA that I'm going to use to log in to the website with and fiddle with some tasks, then this goes in LastPass. You can store it there and use the extension or the vault to help me remember a complex password. You can also use LastPass to send credentials to my team members and then they can use LastPass to log in with.

In the second case, if you have made a script which asks AWS for what instances you have running right now then that script is going to need some credentials to log in with. You could put them in LastPass if you wanted but every time you went to run the script it would mean you have to go and get them from LastPass, enter them into the script somehow and get my list of instances.

Vault however presents an API which your script can use to ask for credentials when it runs. The credentials are stored in a secure way by Vault and you can now store much more in there as well.

The above is also why Vault wouldn't be useful for most people to do what they want to do. You don't want to have to write something to talk to Vault and create browser extensions etc when LastPass is, kind of, affordable and good at what it does.

I hope this helps!