docker run vs docker-compose - different results?

I'm a bit new to Docker as well, but are you 100% certain that you performed a Docker push and your commit of your docker image that you're building with composer? And you have matching versions on the remote and your local machine?
I'm also not seeing where you are mounting the '~/.guacamole/lib' path in your docker-compose.yml under volumes:


Consul: how to remove dead client nodes quickly

As a best practice, you should gracefully deregister the node gracefully. In this case, consul will know the the node has left and will be removed from the cluster. Otherwise, the consul cannot distinguish between a temporary failure, agent crash, network partition, etc.

There is a github issue as well related to this.

Quoting few important points from the issue.

  The nodes should automatically reap out after 72hours (not yet configurable, but soon). Otherwise, the best route is to issue a graceful leave before destroying the nodes (consul leave), so that they can be reaped immediately. They are kept around for that long since without a graceful leave, Consul cannot distinguish between a temporary failure, agent crash, network partition, etc.

.

  force-leave should push them into the "left" state. Nodes are not reaped until they are in failed for 72h or in the left state. "force-leave" just moves a node from the "failed" -> "left" state. They are not removed from the members list for 24 or 72h.


What are Pros and Cons of various VCS and CI/CD tools to store, change and deploy configuration snippets of more than 1000 customers in a VCS?

Personally I'd keep them in separate repositories, for clear customer isolation:

minimal/no risk of unwanted interference between customers
different access control for different customers is possible
different CI/CD pipelines and/or configs for different customers is possible
simpler/standard CI/CD configurations
clean per-customer repository history
easier calculation of per customer resource usage
lower maintenance risks - better to have just a few customers down than all of them at once

I see some cons as well, but IMHO not enough to tip the balance:

more resources needed (one CI/CD pipeline per customer), but these costs can be reflected in the service pricing. And they should get a lot smaller due to automation.
sharing of configs/procedures across customers becomes difficult

If a unified VCS vision/presentation is desired I'd still keep separate per-customer repositories (thus keeping some/all of the pros above), collectively managed under a single umbrella repository:

the umbrella repository would contain just a manifest file listing the per-customer repositories, their versions and maybe some other relevant info
changes in one or more of the individual repositories need to be accompanied by corresponding manifest changes in the umbrella repository to advance the overall project
a small set of wrapper tools would be used to operate at the umbrella repository level but controlling the individual repositories


What are best practices for running Hashicorp's Vault in terms of high availability?

According to this documentation two vault servers have to be run. One of them has to be active and the other passive. In order to run such a setup both vault servers need to be unsealed.


In short, I see two categories of tests for your infrastructure: 1) does it have everything you need to run your application and 2) does it not have any superfluous stuff.

First and foremost, you can treat the test suite of your actual software as a kind of "meta test" for your infrastructure. As long as you create the infrastructure from scratch for each test run, and the test suite runs completely on that infrastructure (i.e., doesn't use outside services) the fact that the whole suite is green means that your codified infrastructure is sufficient as well.

Second, especially from the security perspective, you can write tests against your infrastructure. I.e., if one part of your infrastructure is a VM running Linux, you might write a test that does a port scan against that VM, to make sure that there are no unintentional ports open, which may have been installed by an unintended apt-get install side-effect. Or you could write tests that checks whether any unexpected files were changed after your proper test suite has completed. Or you could check the ps outputs of your VMs or Docker containers for unexpected processes and such, build white-lists etc., and thus get automatic notification if some 3rd party package changed in an undocumented (or unnoticed way) in some upgrade.

These second type of tests are, in a way, similar to what you would do in a classical ops setting anyway, i.e., hardening your servers and checking for intrusions, avoiding full ressource and such.


How do I prevent service A from being placed in the same node as service B in Docker Swarm?

At present the answer is "no", swarm mode does not support affinity or anti-affinity between a service and other containers/services running on a node. However, depending on your goal there are other options.

One option is to reserve resources like CPU and memory for your container. This will ensure two resource intensive applications do not schedule themselves on the same node if there are insufficient resources.

The other option, as you've mentioned, is to use labels and constraints. You can give the same label to multiple nodes, which is often done to specify a type of node, or assign nodes to a specific team/project/environment.

Both of these options will reduce your fault tolerance, particularly if you assign a label to one or a small handful of nodes. If those nodes with the label go down, docker will be unable to reschedule your service on another node.


Jenkins Pipeline steps that run before Jenkinsfile

I see (at least) two possible solutions that both have their drawbacks.

Using a shared library containing all the code for your static analysis. That code can then be called as a step in the pipeline.  https://jenkins.io/doc/book/pipeline/shared-libraries/. This doesn't keep the execution out of the pipeline completely, but it keeps the code out of each pipeline/branch and is (in my opinion) a pretty straight forward solution.
Putting the static analysis into a separate job, trigger that job for example with a git hook and then let that job trigger the pipeline. I my opinion a more complex solution that has more moving parts, but may be closer to what you're asking for.


How to communicate in a multi-organization spanning project

Establish a wiki (such as Confluence, DokuWiki, etc).
Establish a chat/communications channel (such as through Slack or Mattermost).
Share some code repositories (such as through GitLab, Bitbucket, GitHub).
Share some binary repositories (such as through Artifactory, Nexus, Pulp).
Occasionally send out a "newsletter" email that showcases what your team are trying to accomplish.


How to Integrate Amazon API Gateway with EKS using Application Load Balancer (ALB)?

From the same page:
The steps to integrate ALB and NLB with API Gateway are identical.

Which follows by a sentence with a link to a documentation for an ALB ingress, including prerequisites for VPC subnets and EKS resources.
Yes, you have to create one unless you'd want to configure ALB manually instead of using Ingress manifests inside k8s.


What does implementing Hashicorp Vault mean in practice?

In the hope of resurrecting this question, I will give my opinionated answer.

As I understand it, the use of Vault is part of the drive to give more autonomy to teams, by removing fear of leaking or changing secrets. By providing somewhere to store and manage secrets, ops can give dev the means to continue developing at their own pace, following the practices of continuous integration and working in small batches.

With a product like Vault in the pipeline, secrets can be consumed safely in the various phases - build, test, bake, deploy, etc. Implementing Vault in practice means less personal interaction with the service, but more machine interaction with the service, from trusted components in the pipeline.


Can I move SSL certificate files from one server to another for the same domain?

Your listen 443 need to have ssl parameter to tell Nginx that all connections accepted on this port should work in SSL mode.
See example at https://nginx.org/en/docs/http/configuring_https_servers.html#single_http_https_server
Shown here for convenience:
server {
    listen              80;
    listen              443 ssl;
    server_name         www.example.com;
    ssl_certificate     www.example.com.crt;
    ssl_certificate_key www.example.com.key;
    ...
}


What is a difference between a Walking Skeleton and an MVP?

The walking skeleton reflects the rough structure/shape of the product, but it may be entirely stubbed out (initially), meaning it may not be actually functional. The skeleton cannot move without the muscles.

The MVP however must meet a certain minimum functionality level to be considered viable.

As development progresses the walking skeleton typically morphs into an MVP and eventually into a final product.


Github rest api post label gives error for array input

From my understanding of the docs, you should directly send an array (which is valid JSON), without wrapping it in an object. That is ["bug"] instead of {"labels": ["bug"]}.

So, the proper request should be:

curl -X POST -u githuser:gittoken \
    https://api.github.mycompany.com/repos/team/repo/issues/560/labels \
    -H "Content-type: application/json" -k \
    -d '["bug"]' \
    -H "Accept: application/json"


How to make the Jenkins agent build then start the server?

You are running into a conflict between jenkins nature of making sure everything is done before a job is complete and your desire for some long-running things to be asynchronous.  There is no right answer here, both are reasonable desires, but you and jenkins are pulling in different directions.  It may be ok for some parts of the jenkins job to not be completely done but they are good enough for the job.  This is a decision you get to make.

If getting this working inside of jenkins is too much trouble there are work arounds.  You can make your own daemon that looks for job completions (via files or database entries) and then starts something else.  Obviously this means you lose the UX features of jenkins that let everybody know where something is.  But you could make this work inside of jenkins by having different jobs also.  The quick job finishes and then the longer job takes over.  Since each is its own job it can have timeout requirements that make sense for it.


Github commit message when deploying to aws beanstalk in Jenkins

This gives a short commit message

git show -s $GIT_COMMIT --format="format:%s"

I'd probably put this in the "Excute shell" command box

Hope this helps!


What branching strategies/workflows are available for Git repositories?

The Atlassian website includes a comparison of different workflow strategies available when concerned with Git. More strategies exist, like the one used by Linux Kernel team, but are not relevant to most organisations.

The workflow types most commonly used are -

Centralized - every change is added using a single branch.
Feature Branch - every change is added on its own branch and considered for merging into a central branch for integration.
Gitflow - based on feature branch workflow, but also adds another branch for stable releases, and another bunch of branches for hotfixes on that stable release branch.
Forking - in OSS projects often the whole project is copied over to an additional repository and changes are merged between multiple public repositories, or not merged, but still available to the public.

Depending on the organization, one of these might be more suitable.

Centralized - non-restrictive, allows everyone to shoot anyone to "break the build" by pushing changes, least overhead and process involved. The advantage is a continuous integration of changes, even when those don't work.
Feature branch - allows anyone to add changes by adding some overhead process, but integration into main collaborative work is a controlled process. Has the disadvantage of integrating changes late, when requests to merge are forgotten or not attended.
Gitflow - huge overhead in managing the various branches, causes a lot of confusion for regular people who just want to get their changes into the branch. Can work well when most developers only see the "feature branch" part of this. Merging between the development and stable branches often happens late, and integration is not trivial. Adds a lot of extra overhead and terminology that everyone must understand well.
Forking - the wild west, requires a server that supports managing multiple repositories and repositories per-user (GitHub, GitLab, BitBucket Server, etc...). Allows most freedom for each contributor to do whatever he wants, but does not force integration. Demands that integration is done piecemeal in small bits and very controlled, usually by using pull requests across repositories. Works well in unorganized teams such as OSS project contributors.


Can we talk about "reproducible builds" when working with interpreted languages?

A "build" does not have though to be a compilation step.

".. the term build is similar to that of any other field. That is, the construction of something that has an observable and tangible result."
https://en.wikipedia.org/wiki/Software_build

More important in this context is to be able to reuse results of your build in other contexts to build new things.

That is, also in PHP and Javascript you could produce some packaged artefacts, be it a zip file or language specific archive linkable to a dependency manager: NPM in Javascript and Composer in PHP for example.


Jenkins managing a pool of servers across restarts

I found the missing piece in the Lockable resource plugin.

lock(label: 'server-pool', quantity: 1) {
   ...
}

Now the servers to be used for testing don't need to be managed as slaves at all and that greatly simplifies things.


Running Jenkins in docker with custom uid / gid

As usual I think I came up with the answers shortly after spelling out the problem carefully here. I would be great if someone could confirm that my understanding is correct.

TL;DR: It's not possible to use --build-arg for FROM images

The FROM command basically tells docker what binary image to start from. The FROM image is not built recursively. This obviously means that --build-arg won't be taken into account in the FROM image.

So why is uid and gid even ARGs in the first place if I can't use them?

It is possible to use them in a reasonable straight forward way. docker build accepts git repo URLs when building, so you can indeed kick off a Jenkins docker with custom uid and gid quite easily as follows:

docker build -t my-jenkins --build-arg uid=1003 --build-arg gid=1003 https://github.com/jenkinsci/docker.git
docker run -it -p 8080:8080 --rm my-jenkins

But I need to create my own Dockerfile :-(

If you really need to custobmize the Jenkins image and need a custom uid / gid, the best way forward is to change the uid / gid of the Jenkins user in your Dockerfile. The uid = 1000 and gid = 1000 is already hardwired when your part of the Dockerfile kicks in.


Service Level Agreement

https://en.wikipedia.org/wiki/Service-level_agreement

https://www.atlassian.com/it-unplugged/best-practices-and-trends/stop-hating-on-slas

https://www.cio.com/article/2438284/outsourcing/outsourcing-sla-definitions-and-solutions.html

What will happen if the docker registry is down? Are all departments blocked? What is the impact?

If all production images reside in a docker registry and someone wants to release a newer version to production, but the registry is down, is it still possible to release the product?

SLA is about defining processes, checking the impact of each of them and the more important the higher the SLA.

One could this tool to choose a certain SLA for a certain procedure:

https://uptime.is/

A SLA of 99.999 https://uptime.is/99.999 means a yearly downtime of 5m 15.6s, while https://uptime.is/99.9 means 8h 45m 57.0s.

One could set a SLA of 99.999 for the docker processes, but is a downtime of 5m 15.6s really required or is 8h 45m 57.0s in a year acceptable? If one hosts an ecommerce website and the webshop is down a couple of hours this will cause a lot of harm for the company as customers could not buy products and will probably buy products somewhere else.

For the docker processes that are used by developers and for releases to production one could argue that the first one could be 99.9 and the latter 99.999.


As per https://mesos.readthedocs.io/en/0.22.2/attributes-resources/ states, attributes are separated by ';' not by ','.


This command will give you the pod CIDR addresses for each of the nodes in your cluster.

kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'


Yes, docker containers when they are running can take up some space as per your workload and application but as mentioned in the other answers - containers are built over layered filesystem therefore only extra space they consume is from some file/object they create/download during runtime which is not the part of base image like some debug log files, etc.
For ex. - When you run 100 containers of a base image of size 1GB, docker will not consume 100 GB of disk space as it leverages layered filesystem and top immutable layer is shared across all containers of same image.
When you stop and remove (yes!, remove explicitly or use --rm flag with docker run command) a container then it's space is reclaimed.
To check space used by containers (and other objects), you can use:
$ docker system df and for detailed info. use verbose flag $ docker system df -v
Initially when you start a container the SIZE column will show 0B but as soon as you start writing to the disk SIZE will change.
EDIT: added more info. as suggested by @Benjamin in comments


In Azure Devops and TFS, is it possible to create a build artifact and release definition in a different collection than the original code repository?

Since on-premise TFS and the newly renamed Azure DevOps Server 2019 still have the concept of collections, which is the highest level of separation of data you can have (separate physical databases), you'd need to treat a code repo in Collection 1 as an external repository rather than try to discover it as a known internal repository.

You can do this in the Build pipeline of a build in Collection 2 by selecting External Git (if your repo is Git, TFVC is not supported), then clicking Add Connection and entering the Git repository URL and login credentials like in the screenshot below.  Once connected, your pipeline, if it produces artifacts, will store them in Collection 2, which can be used for a Release pipeline.


Deploying a Kubernetes cluster for production on multiple clouds

At work we are currenly using a multi AZ Kubernetes cluster on AWS and we're using kops along with Terraform (kops generates the Terraform configuration files) to provision the cluster. What is not clear to me is if your intention is to run a single multi-cloud cluster or if you want to run multiple clusters in multiple clouds.

Anyway, our current setup is a multi-master highly-available multi-AZ Kubernetes cluster. I'll try to explain step by step.

The first thing in order to create the cluster is to generate the Terraform configuration with kops (you could directly apply the changes in AWS by directly using kops, but in our case we think it's best to keep the Terraform files versioned in git, to be able to change details about this configuration and inspect them, also). For instance, this could be the command we used:

kops create cluster \
--name=my_new_cluster \
--ssh-public-key=k8s.pub \
--dns-zone=<your-name>.<your-domain> \
--cloud=aws \
--master-size=t2.medium \
--node-size=t2.medium \
--vpc=<vpc-id> \
--zones=us-east-1a,us-east-1b,us-east-1c \
--master-zones=us-east-1a,us-east-1b,us-east-1c \
--out=my_new_cluster \
--target=terraform

This creates the Terraform configurations (in the directory specified by the --out flag) for a Kubernetes cluster which nodes and masters are distributed along 3 Availability Zones (us-east-1a, us-east-1b and us-east-1c). This implies creating:

all necessary subnets, security groups and route tables
IAM roles
AWS Auto Scaling Groups to manage the cluster nodes and masters (it's the magic of all this, they're the ones that maintain the high availability of the cluster by launching extra nodes as traffic increases, and replaces faulty ones)
all EC2 instances and EBS volumes
Route53 DNS records to refer to the cluster API and internal DNS for k8s to communicate with pods.

kops uses kubectl internally to create the cluster and all necessary configurations in Kubernetes.

After that, you could terraform apply the configurations and watch it all go up in AWS. It's quite simple, really, but you have to learn how Terraform works. I've found it very intuitive.

Later on, you may need to edit you cluster. There's a command: kops edit cluster <cluster-name> that let you do that. It's all configurations in  yaml format. I won't get into specifics but after some changes you may need to kops rolling-update <cluster-name> to apply them.

Hope I've been clear.

Regards.


Apache Mesos - cases pro and contra containers?

Your impression is false, linux containers (LXC) exists since 2008, mesos and docker have started later. All three makes use of cgroups available since kernel 2.6.28.

For your overall question, mesosphere have a blog post about it.

But mainly you're looking at it from the wrong point of view, choosing the underlying orchestration system should not be done from 'how to deploy this application?' point of view, but more on 'What would be the best orchestration for what I have to deploy as a whole ?'. Making the choice between swarm/kubernetes/mesos/k8s on mesos/swarm on mesos/whatever is very dependent on your environment, your existing knowledge, your team skills etc. and we can"'t really help you choose without an extended description on what you're willing to manage.


Azure DevOps service connection: lifetime of service principal

I will try to elaborate a lite about those topics.

When secret expires and you got error, you have to refresh connection. Go to Edit dialog for connection and save without any changes. Described here: Azure devops service connection expired and cannot edit/renew

But I have already tested scenario when I delete all secrets assigned to service principal used for the Azure DevOps connection. And it is still working. So I can assume, that it depends on how you create connection (automatic or manual service principal). When created with automatic path, it doesn't depend on secrets.

On the other hand, when I add connection with manually providing service principal, then delete all secrets, the connection stops working.
Yes, the service principal will exist. But best practice is to set up more then one Owner of the service principal. If the only one service principal owner is deleted, you need to have higher permission to manage all service principals.
Service principal will not expire itself, only secrets generated for it. To delete service principal object, you have to do it manually.


Ansible: Test if multiple RDS instances are all stopped

If I understood you correctly, you need free strategy.
It will run all tasks ASAP, not waiting for anything.
https://docs.ansible.com/ansible/latest/plugins/strategy/free.html

Bear in mind, that logs will be a mess :)


Is openstack public cloud like Google drive, aws where we can store data?? Or something else?

Openstack is something you might install to setup an self-service IaaS platform that manages VMs, storage and network. Whereas the top three cloud providers have written their own proprietary IaaS platforms a small cloud provider, or a large company, cannot feasibly code there own cloud-like management software, so they might setup their “private cloud” using openstack (or, say, a big vendors distribution of it) so that many internal projects can use it as self-service IaaS. To quote Wikipedia with some expansion by myself:

  OpenStack is a free and open-source software platform for cloud
  computing, mostly deployed as infrastructure-as-a-service (IaaS),
  whereby virtual servers and other resources are made available to [internal or external]
  customers. The software platform consists of interrelated
  components that control diverse, multi-vendor hardware pools of
  processing, storage, and networking resources throughout a data
  center. Users either manage it through a web-based dashboard, through
  command-line tools, or through RESTful web services.

People might talk about it being “your operating system for private cloud” which is a bad analogy. In general the term “operating system” is often deliberate misused as it has positive connotations and might indicate to none technical people the broad benefits of investing in a given technology. For example people might also talk about container orchestrators such as Kubernetes or Docker Swarm being “the new operating systems of the cloud” which is a similar poor analogy.

If a backup project said it provides connectors for AWS S3, Azure Blob, Google Cloud Storage, and Openstack Swift then it would be more clear exactly what it does: write large binary backup files into a cloud store. Such a product could then accurately say it supports AWS, Azure, GCP, and Openstack. Just to be clear I am not saying the the technology you are looking at actually uses those APIs I am making a general point about openstack having feature parity with many public cloud technologies.


Permission denied within mounted volume inside Docker/Podman container

Thanks to the people here, the solution is quite simple (but not obvious):

My GNU/Linux container host has SELinux activated, and that's why I was having permissions problems. The solution is to simply append a :z to the podman run volume argument so that this:

podman run -it -v /host/foobar:/src_dir /bin/bash

becomes this:

podman run -it -v /host/foobar:/src_dir:z /bin/bash

That's it.


Reuse Alertmanger Slack Integration into Jiralert template

In the end solved it myself, I had to just define jira.summary and description respectively and copy/paste the contents from the slack integration in the given fields.

{{ define "jira.summary" }}]
    {{ .Alerts.Firing | len }} {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
    {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
    {{" "}}(
    {{- with .CommonLabels.Remove .GroupLabels.Names }}
        {{- range $index, $label := .SortedPairs -}}
        {{ if $index }}, {{ end }}
        {{- $label.Name }}="{{ $label.Value -}}"
        {{- end }}
    {{- end -}}
    )
    {{- end }}
{{- end }}
{{ define "jira.description" }}
    {{ with index .Alerts 0 -}}
    :chart_with_upwards_trend: *<{{ .GeneratorURL }}|Graph>*
    {{- if .Annotations.runbook }}   :notebook: *<{{ .Annotations.runbook }}|Runbook>*{{ end }}
    {{ end }}
    *Kubernetes Cluster:* `default`
    *Prometheus Alert Details*:
        {{ range .Alerts -}}
        *Prometheus Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
        *Prometheus Alert Description:* {{ .Annotations.message }}
        *Prometheus Alert Details:*
            {{ range .Labels.SortedPairs }}  *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
    {{ end }}
{{- end }}


Is it possible to trigger an AWS Lambda function directly from CloudWatch alarm?

As far as I know, it is not possible. The the offcial AWS Lambda documentation says it support CloudWatch Events, but no mention of CloudWatch Alarms (and CW Events != CW Alarms)

What you are currently doing, sending the alarms on a SNS and using listening on a topic seems the way to go. In fact, CloudWatch Alarms only outputs to SNS so far.


Ansible Dynamic Inventory - generate inventory with ip addresses under specific tag with underscore instead of dot

It works if I comment out the hostname_variable, because it allow to override the inventory_name with an ec2 variable, instead of using the destination_variable

#hostname_variable = private_ip_address
#hostname_variable = ip_address
destination_variable = public_dns_name
vpc_destination_variable = private_ip_address

For Private IPs:

destination_variable = private_ip_address


Can Jenkins master connect to a slave node in a private subnet?

As described in Jenkins documentation https://jenkins.io/doc/book/managing/security/#jnlp-tcp-port, you need to configure:

Jenkins to use fixed TCP port for JNLP agent, for instance 39057
AWS loadbalancer to listen on port 39057 and forward it to Jenkins on same port


I searched for the healthcheck/postgres:alpine and I found the following repository with the Dockerfile:

https://github.com/docker-library/healthcheck/blob/master/postgres/Dockerfile.alpine

Assuming this is the correct repo for your docker image, the dockerfile specifies:

FROM postgres

COPY docker-healthcheck /usr/local/bin/

HEALTHCHECK CMD ["docker-healthcheck"]

So it extends the postgres docker image which is documented here:

https://hub.docker.com/_/postgres

Specifically for the password they mention the following:

  POSTGRES_PASSWORD This environment variable is recommended for you to
  use the PostgreSQL image. This environment variable sets the superuser
  password for PostgreSQL. The default superuser is defined by the
  POSTGRES_USER environment variable.

  Note 1: The PostgreSQL image sets up trust authentication locally so
  you may notice a password is not required when connecting from
  localhost (inside the same container). However, a password will be
  required if connecting from a different host/container...

Read the whole documentation at the image homepage.


What you're looking for is a CI/CD pipeline.  There's lots of articles around it, but the main steps would be as follows:

Some form of build artifact (it can be a docker container, but it can also be a tarball, or just code in a git repo).

Some way to automatically deploy your code.  This is done by using a CI/CD tool (literally any of them will work fine, but for the love of god, don't use Jenkins, you'll thank me later).

For something as simple as how you describe your app, you can just use a CICD tool to SSH in and deploy to your existing server.
For example, CircleCI (one of my favourite tools and what we use at my current job), has a free tier that gives you 2500 minutes of build time a month.  It also has an SSH orb you can include in your pipeline configuration that will SSH into your server and run a few commands to pull down the latest code and restart your server.
Here is a good Getting Started guide that should get you up and running with Circle.
If you'd like to start using docker, here is a good guide on how to run your Django app with it.  From there, you would need to set up a Dockerhub account and push your docker image there, to be deployed on a server.  Note you'll have to make sure you can configure your application through environment variables.  It's never a good idea to have an artifact with "baked in" configuration, since this method doesn't scale in any real way.


How can container user and host user both have access to same files in volume?

I was tripping myself up: when I was trying the --user flag, I still had --userns-remap set. Running the daemon without --userns-remap did the trick.


Bad practice having a single SSH gateway with access to all environments?

I assume your users connect to the gateway with ssh (possibly with additional credentials like a hardware-generated one-time-password), and then jump to the actual servers with a second ssh connection.

In itself, this is fine (and the usual thing that's done regularly by companies I've worked with). The secondary ssh connection (still) takes care of only letting those people in who are authorized. The job of the gateway is primarily to avoid the need for the ssh port of the servers to be open to the internet on the network level.

Obviously, you want to avoid people opening ssh tunnels, using proper settings in your global ssh/sshd configuration, and you will want to tighten down the gateway itself (for example, by having read-only home directories for your incoming users; and having no tools installed that can help in somehow bypassing any of your rules - i.e., no telnet, netcat and so on; ideally a chroot environment in which nothing except the ssh executable is available).


ansible task -easy_install with requirements.txt file

You can loop with dictionaries using loop. It doesn't look like the easy_install module handles versions with a module attribute, so you might have t pass the version on the name attribute.

- name: install things with easy_install
  easy_install:
    name: "{{ item.name }}=={{item.version}}"
    state: present
  loop:
    - { name: 'pip', version: '18.1' }


You can certainly do this through a handler or task, but first you need to generate the trigger conditions. The swap space is in the Ansible facts that are returned when setup: is run. From the Ansible docs:

  Facts

  These are variables that contain information pertinent to the current host (inventory_hostname). They are only available if gathered first.

  ansible_facts
     Contains any facts gathered or cached for the inventory_hostname Facts are normally gathered by the setup module automatically in a play, but any module can return facts.

  ansible_local
     Contains any ‘local facts’ gathered or cached for the inventory_hostname. The keys available depend on the custom facts created. See the setup module for more details.

As an example:

ansible -i localhost, all -c local -m setup -a filter=*swap*
localhost | SUCCESS => {
    "ansible_facts": {
        "ansible_swapfree_mb": 437,
        "ansible_swaptotal_mb": 979
    },
    "changed": false
}

In your playbook your task to reset swap should then have the conditional like:

when: ansible_swap_free_mb < 500

In your case, however you want to trigger on swap used, so you would have to compute the difference between total and free.


Nexus 2 was deployed instead of Nexus 3. The docker feature seems to be introduced since Nexus 3.

This blog indicates how to configure a docker repository and how to push and pull docker images.

The blog contains multiple pictures. The following picture indicates how to enable the docker registry:


What is a 'Feature Flag Toggle' and when to use them (or not)?

Without repeating the content of https://martinfowler.com/articles/feature-toggles.html, since it is an amazing in-depth explanation on what feature flag toggles are. I will just focus on the DevOps aspects.

According to the 2014 State of DevOps Report prepared by PuppetLabs, there are four major metrics to measure IT performance:

Lead time for changes
Release frequency
Time to restore service
Change fail rate

These also contribute to organisational performance overall. So it means that if your IT is doing great on these metrics, your bottom line gets more $$$.

Continuous Delivery is enabled by these metrics, and has been described in depth in the book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble.

In context of Continuous Delivery, there is an important distinction that differentiates it from Continuous Deployment. And that is the decision when to do a release of features (to customers).

Keeping changes smaller in size, and deploying (copying code) half-baked features to production systems with a feature flag toggled off allows to shorten lead time for changes.

When features are finally finished, doing a release is a decision left to the business. Maybe a release of a new feature needs to be aligned with some marketing, or a release in another part of the business like a feature in the mobile app.

Features can be released using A/B experiements to only a part of the customer base, or to specific people, or even direct to general availability (GA). Although releasing to GA is often done only after there is enough certainty that the feature works as expected. One might argue that this in effect affects release frequency to be higher.

This decoupling of release and deploy is almost impossible to achieve without feature flag toggles.

Naturally when no deployment is required to toggle a feature off, then the time to restore service is lowered substantially.

And by using feature flags that release features to a small slice of the customer base, the change fail rate metric can be improved significantly as well.

So a simple mechanism called feature flag toggles enables much better IT performance, and in turn improves organisational performance overall.

Great example of how this is done in real companies can be found at Flickr (on of the earliest public posts on the subject), and at Etsy. But many others have adopted the practice and talked about it in length, for example the famous engineering culture at Spotify videos.

Etsy are showing off their internal tool to manage feature flags, called Catapult, in multiple presentations found around the web. And Intuit release an open-source tool called Wasabi that helps manage feature flags.


How do I access the web UI for Kubernetes clustered that were created via EKS?

With EKS, the Kubernetes UI need to be deploy manually. You have a fully detailed documentation on how to deploy Kubernetes UI in EKS.


Where should I store my build configuration files?

Normally the build configuration needs to stay consistent with the product code, so it should be stored into the same git repo as the product code.

It's true, the file would get merged when a branch merge from the parent is performed.

But often it's a trivial (empty/fast forward) merge - if there were no changes to the msbuild xml file in the parent branch.

And if the merge is not a trivial one it means there were changes in the build process in the parent, which need to be taken into account in the child branch as well since it also picks up the code updates that go with those build changes. In other words the msbuild xml merge is required in such case to avoid build breakages.


Why should you do port mapping for docker containers?

TL;DR

There are a number of use cases for port mapping, but for DevOps at scale the primary reason is generally to enable mapping well-known service ports to available ports on the host. This matters when you're running large numbers of containers that use the same port by default, and you don't want to manually assign or track alternative port numbers.

A Very Short Port Primer

As a general rule, a port can only map to a single service or process on each host (multiplexing ports and multi-port services are something of an exception). Only 65,536 ports are available for services to bind, with the lowest 1,024 generally reserved for binding by the root user. Services also typically bind to well-known ports such as 22, 53, or 5432 to make the service easy to find. All of these issues matter, but it's often the last issue that typically concerns Docker hosts most.

Mapping Container Ports

Imagine that you have multiple PostgreSQL containers on a single host. By default, each wants to bind to port 5432 as its default. While you could certainly modify each container or run-command to bind the container's service to a unique host port, this quickly becomes a hassle at scale.

Instead, Docker and other container managers make it easy to map ports between the host OS and the container. For example:

# launch three PostgreSQL instances
for i in {1..3}; do
    docker run --rm -d -P postgres:alpine
done

# show port mappings for each container
docker container ls -q --filter="ancestor=postgres:alpine" |
    xargs -n1 docker port
5432/tcp -> 0.0.0.0:32773
5432/tcp -> 0.0.0.0:32772
5432/tcp -> 0.0.0.0:32771

This shows that you have three instances of PostgreSQL, all happily listening on the default port of 5432 inside their containers. However, each instance is listening on a different port (32771, 32772, and 32773) on the Docker host!

At scale, you would typically use DNS, autodiscovery, linking, or container networking to help clients and applications find the right PostgreSQL instance to connect with. With just a few instances running, parsing docker ps may be sufficient for your needs. Your specific use case may vary.


Automatically write new bash history to change-log file

Try using GNU Screen or Tmux. These are called "screen multiplexers" but you'd get a number of benefits

The commands you run will continue running even if you disconnect, or you have network related connection drops. Both software allow "re-attaching" to the running session
Command outputs will persist in memory. You can start a long running command, log off, then connect back some time later and see the output of the command.
Save the whole shell session to a file. You can have the log of all the commands you issued, as well as the output you got, in plain text format.


Putting it behind a DNS name with a load balancer across multiple SDS servers is a good setup for reasonable availability. If SDS is down, Envoy will simple not get updated, so it's generally not the most critical failure -- new hosts and services simply won't get added to the cluster/endpoint model in Envoy.

If you want higher availability, you set up multiple clusters. If you add multiple entries to your bootstrap config, Envoy will fail over between them. You can either specify multiple DNS names or multiple IPs.


Cannot understand jenkins deployment error

I resolved the problem publishing the ivy.xml file to nexus too, adding publishivy="true" to my publishing task in my build.xml file.


Are there automated quality checking tools for Ops code

It's difficult to provide a definitive answer given the scope of the question, but the  goal of testing infrastructure is both laudable and seldom done at all.
To my knowledge, you are looking for something akin to Open Policy Agent which provides a general policy language in which to express arbitrary policy as code. I would suggest investigating that at length.
At the risk of not providing a complete answer, I will give my best shot for some concrete examples as well.
Compliance as Code
In general, you need something which is often referred to as a "compliance framework", and in the DevOps world more often referred to as "Policy as Code". This allows you to define policies as code, which really means "machine-readable and executable". You specify what the policy is in the DSL of the framework, it is executed at some stage of your pipeline, and fails the change if it violates the policy.
To illustrate the point, consider someone submitting a pull request to a terraform module that inadvertently opens a port on a security group to the world which the policy says should be closed. This may be difficult to determine when you're writing the code, but the compliance test will catch it.
There are a few ways to do this, but I will mention two. A notable exception is Pulumi's Crossguard, but I don't have enough experience to speak to that.
Hashicorp Sentinel
In the Hashicorp world, there is the Sentinel Framework. This is an Enterprise product, so may not be relevant to your situation, but it is one of the best implementations I've seen. Importantly, it has a built-in testing framework  which aids in writing good policy code, which is also very expressive. I have used this as inspiration for how to write policy in Chef Inspec, which is the other example I will cite.
Chef Inspec
Chef Inspec is a framework for writing policy as code using rspec-style assertions. It uses a DSL based on ruby and supports major clouds (AWS, GCP, Azure) as well as OS-level resources. It is open source, so you can use it anywhere. You use it to make assertions in individual resources which are defined in the inspec DSL, e.g. "user on an OS" or "AWS EC2 instance". Writing the tests consists of assertions against the actual state of those resources, which makes it quite like a unit test. However, Inspec does not have a built-in testing framework for itself like Sentinel does - although you could use the ruby ecosystem for doing this.

Some notes:

Both of these tools allow you to express your policy as code, and to start extending the test coverage of that policy over your existing infrastructure.
Sentinel will catch violations before they are applied, but Inspec needs to be executed against an already-provisioned state.
Both will do what you required, but depend on your tech stack as to whether they will be appropriate for you
There is no magic bullet

for both of them you will need to write your policy as code. Usually this means taking some form of human-readable documentation and writing it in the DSL of the framework.
On the other hand, this provides an excellent opportunity to collaborate with the security team and do what folks are calling DevSecOps these days.


You can find the IP addresses of the Travis build machines here. If you add the IP addresses of the Travis infrastructure you use to your whitelisted IPs in Cloud SQL it should work. Keep in mind these IP addresses can change in the future though.


Problems with golden images are that you have to store a full system image with overhead of data which is not build-specific, and using conventions for configuration to make deploy process the same for all versions.

TL;DR: Infrastructure as a Code, version-specific storage for data outside the app package.

The problem is easily solved by storing config files and templates, data and installation scripts (including ansible playbooks to manage) inside both code and packages/images, so install and initial configuration would always be version-specific (switching to the same commit you are deploying to have configs you need). This could be done with VCS and current package management software (deb, msi, PKG etc. and scripts for unattended deploy and installation) with ease. To avoid using different images of OS and other environment, storing and using a single golden image for all builds is a standard practice: it provides consistency and you can automate CI/CD pipeline for image update process.

If some data is stored outside (i.e. a lot of current Windows apps outside of Store are installed by running a downloader, then installing app) and you have to keep large number of them (no specific branches only like stable-beta-alpha), then storage should be versioned too (i.e. using version-specific hashes as names) and links in install scripts or artifacts would point to the one version they need. But you also have to keep the register of pairs build:data version, so when there's no need to use specific version, deletion of related data could be performed within the same remove process. Also, you have to decide whether you want to store complete images or use some differential systems (incremental or differential snapshots of files/filesystems), latter is much more complex and hard to manage so it's not used widely AFAIK (with an exception for some DBs, i.e. Facebook stores MySQLs snapshots and binary transaction logs so they could deploy or restore exact state up to certain transaction id).


Count of currently active jenkins agents

You can sufix computer/ with api/json and run some script on this response. Count the computers array or group by some one of his attributes, like idle or offline.

https://www.myjenkins.com/computer/api/json


Wordpress on AWS, how to update only a theme

I'm not terribly familiar with elastic beanstalk, so I'm not sure if my method is at all approachable with your workflow, but I have a similar (non-AWS) workflow, and I can share how I do it.

I've managed to only push theme changes by separating the deployment of wordpress, and its mysql DB (both as a docker containers, in my case), and the deployment of the themes (which is a git repo).

This way I can separate/automate the deployment of wordpress, mysql, and my wordpress theme in a fairly loosely coupled manner.

Updating consists of only performing a git pull to the themes directory -- No other wordpress/mysql interaction occurs, so my plugins stay in whatever state they were previously in prior to the theme update.


You should use of the Retrieve Latest Artifact endpoint. It uses the repository layouts to know what is the version in your artifact path, so you want to make sure those are set right for you. In this case, it's a standard Maven layout, which is the default for Maven repo, so you're all good.


How to setup testkitchen with ansible-provisioning from localhost for different vagrant-vms?

According to this documentation it is possible to install ansible and run a ansible playbook by using the following snippet:

Vagrant.configure("2") do |config|

  #
  # Run Ansible from the Vagrant Host
  #
  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "playbook.yml"
  end

end


What are possible implementations (or examples) of the four-eyes principle?

One of the implementation on code is the Pull Request model (PR)  popularized by GitHub.

The main reasoning behind is that only a small set of maintainers of the product will be allowed to merge code into the release branch. Every new feature/bugfix will happen on a new branch and once done will be defined as a pull request.

This allows to test on the merge from actual release (master) code with the code in the PR in an automatic way (Travis being the most popular for public project actually) and give a first feed back on the code quality. Travis CI (for example) can run on the result of actual master with the  code from the pull request merged into it, so it will fail if the merge is impossible, or if the commands defined in the travisci.yml return a non zero exit code

Once the automatic tests have passed, for the 4 eyes principle, it still requires a configurable number of persons to review and approve the change before it gets merged obviously at least 1 person (who is not the PR author) to enforce 4 eyes reviewing the changes.

There's a wide range of options to merge automatically once the quorum of reviewers is met or to still need a manual merge by a maintainer.

The permissions to review and merge can be separated which helps giving more people the right to "vote" on the merge status while keeping the possibility to restrict who can actually do the merge.


How to setup Azure DevOps Pipelines and Releases with multiple apps

I use separate pipeline for each app
you could use powershell task to restart IIS, but I dont think there is a way to do this without shutting down the website. You can also reconfigure website, so copy new code to location X, repoint website to location X, restart website. Then you can easily point it back if something fails and switch is almost instanteneous
You can use library variable sets (or groups, dont remember). those can be tied to the stage in the release. it might also be possible to scope regular variables, but I've never tested that.


Should the master node's hardware be the same as the worker node's hardware for a Kubernetes setup?

The control plane (master nodes) and the data plane (worker nodes) are completely different workloads. You can use very different hardware or virtual machines if you want.
You can use some with x86, some other nodes with arm64 or even Windows nodes are supported as worker nodes. But you probably need groups for your nodes, since the application typically only run on a specific type of nodes and it is easier to understand e.g. metrics if the application run its instances on the same type of nodes.


When considering mono-repo vs many-repos you might look at the following questions, which should also affect the composition of the code directory tree structure. There is a slight general trend toward mono-repos.

Discoverability

How easy it is for developers and ops people to discover the code for some feature. Are there clear guidelines on what goes where? Are they enforced somehow through automation? Are new repos automatically linked as submodules into correct location? Does the build or deploy process depend on the tree structure making sure people need to put it there in order for it to be even noticed? What is the size? How long it will take to grep code in certain place? Are there generic lib/ directories where devs and ops end up sharing libraries that create new unexpected inexplicit dependencies?

Deployability

Does any of the code need to be present on machines that you deploy to? Are you expecting to do such deployment through syncing repository or do you have other established deployment mechanisms. In the first case, will other code creep in during the deployments that does not need to be there?

Durability

Will the concept you are using stand the test of time? Are you a one product shop, but maybe in your future there are new products? How would adding a new product affect the code? Will there be new mono-repo for the new product? What about shared code between products? Looking at your company roadmap, how will whatever you do fare 6m, 2y, 5y down the road? Choices like this tend to be hard to change in practice (even though they shouldn't - in theory).

DevOps

You really want to take down barriers between Devs and Ops. So even just having separate mono-repo for ops could create such barrier. The operation and deployment of code, standing up environments, setting up services, it should all be in reach of developers and with explicit expectations to develop such code in coordination with the ops team, not being dependent on someone doing it for them. If things go well your developers follow through with code for months after it has already been in production and have hand in all aspects of the deployment. How easy it is to make commits that contain both product and deployment changes? How can those commits be grouped together? What is the visibility of ops into product runtime code and devs into product management code?


Scanning fs in a Docker container is (much) slower than on host?

Docker on MacOS runs on a virtual machine (docker-machine), and it's known that docker-machine volume mounts may perform worse than what you'd expect. It becomes worse especially when it comes operations concerning lots of small files, like scanning the filesystem.

Official Docker documentation also mentions this, and suggests some workarounds by lowering the consistency guarantees.

See: https://docs.docker.com/docker-for-mac/osxfs-caching/


Can one use lando.dev for a production environment?

From "the horses mouth":

https://twitter.com/DustinLeblanc/status/1220351066178510848

  We don't recommend Lando in production. The container images we use
  aren't optimized for that and we don't test an open-to-the-internet
  scenario. It's exclusively meant for local dev


How to write a dynamic declarative pipeline that contains sequential job inside parallel job

This was covered in an official blog post when the feature to nest stages inside a parallel block was added.

Here is the example copy-pasted from the blog post.  In this example, the Windows and Linux blocks are run in parallel, but the stages within each block are run sequentially:

pipeline {
    agent none

    stages {
        stage("build and deploy on Windows and Linux") {
            parallel {
                stage("windows") {
                    agent {
                        label "windows"
                    }
                    stages {
                        stage("build") {
                            steps {
                                bat "run-build.bat"
                            }
                        }
                        stage("deploy") {
                            when {
                                branch "master"
                            }
                            steps {
                                bat "run-deploy.bat"
                            }
                        }
                    }
                }

                stage("linux") {
                    agent {
                        label "linux"
                    }
                    stages {
                        stage("build") {
                            steps {
                                sh "./run-build.sh"
                            }
                        }
                        stage("deploy") {
                             when {
                                 branch "master"
                             }
                             steps {
                                sh "./run-deploy.sh"
                            }
                        }
                    }
                }
            }
        }
    }
}

EDIT: I missed that you're trying to do this based on arbitrary input.  If you want to do that, you must use scripted Pipelines (or a script nested in a Declarative Pipeline).  Plain Declarative Pipelines don't support that kind of runtime-defined Pipelines.


What do businesses that employ multiple development teams use to implement DevOps in a structured manner?

Large enterprises will commonly adopt a delivery framework, or operating model, that has been built to support large enterprises.  From an Agile/DevOps perspective, which is my area of expertise there are three frameworks of interest:

Scaled Agile Framework (SAFe®) defines DevOps as delivering business value through the successful operation of software in an environment.  This is achieved by giving early attention to the act of "moving into production" and accomplished, in part, by integrating personnel from the operations function within the Agile Teams.
Disciplined Agile 2.X, also known as DAD, defines DevOps as "DevOps is the streamlining of the activities surrounding IT solution development and IT operations." this is achieved by building feedback loops between the development and operations functions within the business.  DAD goes on to describe Disciplined DevOps as the intersection between the following roles: Enterprise Architecture, IT Operations, Business Operations, Release Management, Data Management and Development.
Large Scaled Scrum does not specifically call out DevOps, Release Engineering or Platform Engineering; however, it does describe a methodology for scaling out Scrum in an organisation that is growing fast, integrating a DevOps specialism within these teams is both logical and well understood.

As far as I am aware there are no statistical data about the enterprise adoption of each of these frameworks and whether they have adopted DevOps practices in addition to Agile practices, however from experience many organisations appear to gravitate towards Scaled Agile Framework.

Kristof Horvath discusses the application of DevOps in these enterprise-grade agile frameworks in his article: Scaling Agile in Large Enterprises: LeSS, DAD or SAFe®?.

It is worth noting that adopting an "enterprise-grade" framework is not a requirement in an enterprise, at the end of the day Agile, DevOps and Lean are about adopting the right practices for your organisation, not the most popular ones.


How to make Load balancer to wait for custom program to finish running?

Use the health check capabilities to ensure that your instance is not in service before your custom program is ready.

If you have a classic ELB, you can point to an specific enpoint, for example, /health_check.php that performs the necessary checks in your application to ensure that your custom program is ready. Then, when a new instance is attached to the load balancer, it will perform the health check action that you have configured an expects a right behaviour of it. If not, the health check fails and will continue to perform new health checks in the configured intervals until instance become in service.

On best practice for web servers behind a load balancer, for example, is to add an specific endpoint that checks that all the necessary pieces for your application is running and in service, that is database, storage, caching processes or whatever you new or consider critical for your system.

The official AWS documentation is very useful for this.

Hope this helps


is there a standard or common approach to defining Service Level Indicators?

Whilst not strictly a standard approach, Google has published an SLI Menu and a process for developing SLIs for user journies:

For each User Journey/Data Flow identify from the SLI Menu suitable types of SLI:

Make a decision about how to measure good and valid events,
Decide where to measure the SLI From out of the following: End User, Client-side Instrumentation, Synthetic Clients, Front-end Metrics, Application Metrics or Server-side Logging.

The document then goes on to describe, how you collate all of your SLIs for all of your user journies then walk the journey to look for coverage gaps.  Then finally set SLOs based upon the SLIs either based upon business need or past performance.


Jenkins Pipeline job - remember previous values

Depending on how long you keep builds for, the Rebuild option in the left hand sidebar will allow you rebuild a previous run of the job with the parameters intact.

Simply find a previous build of the job that that particular user kicked off last time, click into it, then choose Rebuild from the sidebar.

Again this will only work if Jenkins keeps enough builds. We save two weeks for a particular job and I have a coworker who saves a link directly to x build of the job, then once he rebuilds updates his link to the new one, etc. This way he can just open the link he has saved, click Rebuild, and he's good to go as long as he's inside that two week window.

I believe you can even choose to Keep this build forever for a couple of your "template" builds, making them more easily accessible.


Spinnaker keeps showing disabled stages in Jenkins pipeline UI

I was not able to find how to hide a stage, however I found one can skip it using stageEnabled [1].
It takes a Spring Language Expression like so:
{
  "stageEnabled": {
    "expression": "false", // you can also use SpEL, e.g. ${someBoolean}
    "type": "expression"
  },
  "completeOtherBranchesThenFail": true,
  "continuePipeline": false,
  "failPipeline": false,
  "parameters": {
    "FOO": "bar"
  },
  "type": "jenkins"
}

Or from the UI editor: Execution Options > Conditional on Expression > Enter false [2].

https://docs.armory.io/docs/spinnaker-user-guides/using-dinghy/#stage-fields
https://spinnaker.io/guides/user/pipeline/expressions/#dynamically-skip-a-stage


Building a Docker container using Gitlab CI Docker runner

Weird Docker. I changed the WORKDIR in the Dockerfile and it worked:

FROM postgres:9.6

WORKDIR /builds/user/
ADD repo/script.sql /docker-entrypoint-initdb.d/

EDIT
The /builds folder does not exist on the host system. I thought that adding it to the runner's config would create a volume container that would be shared between containers. But no. So I removed it from the runner config, /etc/gitlab-runner/config.toml:

[[runners]]
  name = "My Docker Runner"
  url = "http://host"
  token = "xxxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
  [runners.cache]


How to get Github username variable in Azure DevOps build pipeline?

This is not supported by Azure DevOps pipelines yet. Microsoft will provide it soon. Will update this post once they made it available.

Pretty straight forward way to get last commit user in Bash:
git log -1 --pretty=format:'%an'


Enabling Github webhooks for multibranch jobs?

According to the official GitHub Branch Source Plugin documentation, the plugin can automatically configure webhooks for you if you have your GitHub API token configured in Jenkins global settings:

  You can change the build configuration to perform the folder
  computation at different times. The default setting will scan your
  GitHub repository for changes if it has not received any change
  notifications from GitHub within the past day. This scan will catch
  the repositories you’ve added or removed from the managed GitHub
  repository. You can always force the Folder Computation to run from
  the GitHub Organization page.

  While the build triggers are often enough, you can set up webhooks to
  automatically trigger builds when changes are pushed to your GitHub
  repositories. To do this you must have a GitHub login with a token.

  Go to the main configuration settings page, Manage Jenkins > Configure System
  In the GitHub Plugin Configuration section, add a server with your credentials
  If you need a token, generate one with the Additional Actions > Convert login to password and token

  You can also configure this manually through GitHub itself by
  registering the URL provided in the help section of the server config.


jenkins-x returns error 1 when I try to create a cluster

I just experienced the same issue. It seems the eksctl that is being installed upon first use is not working for me. I went to the eksctl site and installed it from there (https://eksctl.io/introduction/installation/)

With the jx eksctl binary :

➜  ~ jx create cluster eks --cluster-name=konsek-cloud --skip-installation=true --verbose
DEBUG: eksctl is already available on your PATH at /Users/<user>/.jx/bin/eksctl
DEBUG: aws-iam-authenticator is already available on your PATH at /Users/<user>/.jx/bin/aws-iam-authenticator
DEBUG: Dependencies to be installed:
DEBUG: kubectl is already available on your PATH at /usr/local/bin/kubectl
DEBUG: helm is already available on your PATH at /usr/local/bin/helm
DEBUG: brew is already available on your PATH at /usr/local/bin/brew
error: exit status 1

With the manually installed binary :

➜  ~ jx create cluster eks --cluster-name=konsek-cloud --skip-installation=true --verbose
DEBUG: eksctl is already available on your PATH at /usr/local/bin/eksctl
DEBUG: aws-iam-authenticator is already available on your PATH at /usr/local/bin/aws-iam-authenticator
DEBUG: Dependencies to be installed:
DEBUG: kubectl is already available on your PATH at /usr/local/bin/kubectl
DEBUG: helm is already available on your PATH at /usr/local/bin/helm
DEBUG: brew is already available on your PATH at /usr/local/bin/brew
Creating EKS cluster - this can take a while so please be patient...
...


How deploy public key to newly created VM on GCP?

As per the link, You can add keys via metadata. You can try the following.

   metadata:
     ssh-keys: "[USERNAME]:ssh-rsa [NEW_KEY_VALUE] [USERNAME]"

Replace [USERNAME] & [NEW_KEY_VALUE] with actual key value.


What is the equivalent of eksctl option "--alb-ingress-access" in a config file?

Yep this worked, or at least I didn't get an error
    $ eksctl create cluster --alb-ingress-access -f config.yaml


Downloading multple files from remote using chef resource

what about something like that:

%w{
        mysql-community-common-5.7.16-1.el7.x86_64.rpm
        mysql-community-libs-5.7.16-1.el7.x86_64.rpm
        mysql-community-client-5.7.16-1.el7.x86_64.rpm
        mysql-community-server-5.7.16-1.el7.x86_64.rpm
}.each do |pkg|
        remote_file "/tmp/#{pkg}" do
          source "https://s3.amazonaws.com/tmp/mysql/#{pkg}"
        end

       rpm_package pkg do
        source "/tmp/#{pkg}"
        action :install
      end
end

another way:

urllist = {
  { 'url': 'http://some.url1/', 'path': '/some/path1/', 'filename': 'some.file' },
  { 'url': 'https://some.url2/', 'path': '/some/path2/', 'filename': 'another.file'}
}

urllist.each do |urlinfo|
  remote_file "#{urlinfo['path']}/#{urlinfo['filename']}" do
    source "#{urlinfo['url']}/#{urlinfo['filename']}"
    owner 'someowner'
    group 'somegroup'
    mode 0755
  end
end


What are best practices for securing connections between CI and artifact managers in the cloud?

As I know for Git and Jenkins configuration we use SSH key authentication and secure token for security purpose.

In your case you can't open all port to all then you can set revers proxy for your all URL using apache or nginx and forward traffic to port 80, So when request come first it's hit the port 80 and internally it will hit to your specific application port.


ansible error: skipping: no hosts matched

The answer is in the warning you get:
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

When you use hosts: all in your playbook, localhost is not matched.
If you want to run the playbook on localhost, you can do one of the following:

Change your playbook to hosts: localhost.
Explicitly provide an inventory file that has localhost.

You can create an inventory file - inventory.yml with the contents:
all:
  hosts:
    localhost

and then pass this inventory file to ansible explicitly like this:
$ ansible-playbook --connection=local -i inventory.yml playbook.yml
You can find more information about inventory in the Ansible documentation.


Given that you mentioned rarely using some k8s resources, you could explore some alternatives:

With gcloud SDK you could create scripts to be executed for the periods you want to reduce GKE nodes (even to zero) during periods you're not using it
With Google Cloud Functions (GCF) you can create some python/Node.js/Go script to access GKE APIs and also execute the operations to stop/reduce your instances. These GCF could be triggered by schedulers configured by Cloud Scheduler. The cost here for such executions will be really low or even zero.


Personally I don't think using CasC itself will have any negative scalability impact.

Fundamentally CasC means the actual config files are not hand-maintaned, instead they're auto-generated following version-controlled rules - which can be a lot more reliable.

But once the config files are generated - the service using them functions just as it dit with manually-crafted config files.

If anything - using CasC should make such large nagios system more reliable - if done correctly CasC should reduce/eliminate the risk of human error in modifying the config files.


How do i specify the path of a Dockerfile using Blue Ocean

From the official instructions on building Docker images with Jenkins Pipeline:

  The build() method builds the Dockerfile in the current directory
  by default. This can be overridden by providing a directory path
  containing a Dockerfile as the second argument of the build()
  method, for example:

node {
    checkout scm
    def testImage = docker.build("test-image", "./dockerfiles/test")

    testImage.inside {
        sh 'make test'
    }
}

I'm not sure if you can create a Pipeline (i.e. Jenkinsfile) with this exact configuration through the Blue Ocean UI, but I would imagine you can.  However, I personally have found using Blue Ocean to create pipelines to be finicky at best, so I would recommend going over a Jenkinsfile created by Blue Ocean and making tweaks and adjustments by hand anyway.  This would give you the opportunity to adjust the Dockerfile location even if you aren't able to do it via Blue Ocean.


Is it ok to have a Jenkins Server without slave node?

https://wiki.jenkins.io/display/JENKINS/Jenkins+Best+Practices

  In larger systems, don't build on the master.

  If you have a more complex security setup that allows some users to
  only configure jobs, but not administer Jenkins, you need to prevent
  them from running builds on the master node, otherwise they have
  unrestricted access into the JENKINS_HOME directory. You can do this
  by setting the executor count to zero. Instead, make sure all jobs run
  on slaves. This ensures that the jenkins master can scale to support
  many more jobs, and it also protects builds from modifying potentially
  sensitive data on $JENKINS_HOME accidentally/maliciously. If you need
  some jobs to run on the master (e.g. backups of Jenkins itself), use
  the Job Restrictions Plugin to limit which jobs can be executed there.

  One master node and no slave node. All their builds run in master
  node. Is this a bad practice?

Running the jobs on the master nodes means that the jobs have unrestricted access into the JENKINS_HOME directory

  Are there any bad consequences for having only one master node?

As the slaves have unrestricted access into the JENKINS_HOME directory this could be unsafe

  Is my setup (one master & one slave) a bad practice?

It is better than only one master as long as all the jobs run on the slaves

In summary, from a security perspective it is a bad practice to run the jobs on the master.


If you can keep one common deployment strategy across both that would be simpler and less effort from the ops side of things.  Kubernetes seems to be a great choice for the cloud orchestration layer at this point in time.  Does Kubernetes make sense for the on-prem side though?  If you're not planning on each on-prem install including 5-10 servers to spread Kubernetes out over then it probably doesn't make sense to use it there.  All hope is not lost.  :)  You could take the same Docker containers that you ran under Kubernetes and run them yourself for on-prem.  You're going to be stuck dealing with service discovery, but if you're only installing on one server that can be solved in some old school way like picking and assigning ports to specific services.


It really comes down to personal preference.

One additional tool you might not be aware of are shared libraries for Pipeline.  These allow you to quickly write custom Pipeline steps or factor out common Pipeline code without writing a Jenkins plugin in Java.

Between multiple Pipeline jobs and shared libraries, there are many ways to split up your job's code and there's no one right answer on how to do it.  One suggestion I have is to determine what steps in your process are "atomic" - that is, what are the smallest group of steps that you envision admins/ops/devs/etc. might need to run on their own?  Each "atomic unit" should then become its own job.

For instance, let's say you're automating a deploy process, and your process in a general sense looks like this:

build
deploy to dev
deploy to prod

Then I would create three jobs, one for each bullet point.  This gives you some nice advantages:

You can re-run each job individually in case something fails instead of having to restart the entire process over from the very beginning.  Deploy to prod failed?  You only need to re-run the deploy to prod job.  You don't need to rebuild or re-deploy to dev.
Each stage in the process is separated out so you can see how often it succeeds or fail.  Some stages in your process may be more robust or fragile than others; this kind of separation gives you insight into that.
This level of abstraction makes it easy for non-ops people to perform ops tasks (if that is something you want/need at your org).  When everything is monolithic, the only thing non-ops people can do is run the entire process from the beginning.  When everything is split up into tiny jobs, you need intimate ops knowledge to know which pieces to run in what order.  By separating your jobs into independent, easy-to-understand, appropriately bite-size chunks, a non-ops person only needs to press a single button to kick off an automated ops process.


Jenkins - restrict which nodes can be selected by user when creating jobs

You could use Project-based Matrix Authorization Strategy for user authorisation in Jenkins.
Works well with Folders plugin, then you have each project in a separate folder and permissions on each folder are set to a team that looks after the project.
Each node/slave can have access permissions specified. For example, for node A you will only list Team A as users/team who can have Build and Configure permissions. Team B for node B etc. It is a pain if you have dozens of nodes though.
If integrated with AD, then groups as they are defined in AD can be used to restrict/give access without having to list all members of the team who needs access to which node/project.
Authorisation strategy is defined in
"Manage Jenkins" -> “Configure Global Security Option" -> "Authorisation" -> "Project-based Matrix Authorization Strategy"
This article describes it to a degree: https://www.thegeekstuff.com/2016/06/jenkins-security/
You could do the same with just Matrix Authorization Strategy, only you won't have projects. You might not need them at all (depends on how many project/teams you have) and then you will only restrict node permissions.


Does load balancer F5 support CasC (Configuration as Code)?

Yes, F5's do support Configuration as Code. Historically, F5 has made an appliance to manage Configuration as Code called "Enterprise Manager" which pragmatically managed client endpoint F5s (LTMs, etc) using the iControl XML API.

They rather quickly discovered this management appliance was terrible and added a more robust REST API to the client appliances for managing devices (LTMs, etc.; also branded as iControl) which is much easier to use and more flexible and then began building a replacement for the Enterprise Manager branded BIG-IQ.

The point being, you can manage that same API using this REST interface. See their tutorial on DevCentral. Usually, you can find the exact REST syntax and calls on the DevCentral site on a per-version basis like this one for TMOS 12.1.0.

Generally speaking, it is NOT recommended that you use an SCF (Single Configuration File) pretty much ever for a couple of reasons. First, it lacks any supporting files, such as certificates and keys for your SSL profiles or scripted (so-called "external" health monitors), etc. Secondly, it plays poorly if  you use partitioning, as partitions are spread over several files in a foldered structure. These don't consolidate well into an SCF file. You would actually be better off to write TMOS scripts. One of the reasons F5 switched from the bigpipe command to the TMOS shell is that it could be scripted where bigpipe could not be easily scripted. But again, the REST API is preferred. SCFs are really a legacy of version 9 of TMOS and haven't aged well and work poorly in version 12. A significant reason for this is due to the changes in the HA peering between V10 and V11 when they changed to a clustered architecture. It really wreaked havoc on the usability of the SCFs.

Puppet actually has a module for managing F5s if you use this configuration management tool and salt has a runner for it - both using the REST API if you use either of these configuration management tools.


How to pass HashiCorp Vault secrets(k/v passwords) to Terraform EC2 instance

The simplest case is to use the Terraform Vault Provider (https://www.terraform.io/docs/providers/vault/index.html) - which, caveat, I've not used myself.

Example from the Terraform site:

provider "vault" {
  # It is strongly recommended to configure this provider through the
  # environment variables described above, so that each user can have
  # separate credentials set in the environment.
  #
  # This will default to using $VAULT_ADDR
  # But can be set explicitly
  # address = "https://vault.example.net:8200"
}

resource "vault_generic_secret" "example" {
  path = "secret/foo"

  data_json = <<EOT
{
  "foo":   "bar",
  "pizza": "cheese"
}
EOT
}

Optionally you can write a script in whatever language you are comfortable with to make the call out to Vault and invoke via local-exec in Terraform.

Example of invoking a simple command from the Terraform site:

resource "aws_instance" "web" {
  # ...

  provisioner "local-exec" {
    command = "echo ${aws_instance.web.private_ip} >> private_ips.txt"
  }
}


How do I automate deployments with Kubernetes?

As of you are using Rancher the easiest way would be to register custom Rancher Catalog and create an item for each stack/service you want to deploy.

Rancher Catalog is a Git repository with the following structure

Then in Jenkins you can create a job that would call Rancher REST API to deploy/update stack/service.

In its turn Rancher would pull latest version of the Docker image for this service and deploy it corresponding to Docker Compose file from the catalog.

Pros:

generic approach that might be used for almost every app
Jenkins itself might be deployed into Rancher environment, agents might be created in Kubernetes cluster

Cons:

development team should follow solid release strategy to be able to use generic builds
storage drivers are still under the question in alpha release of Rancher 2.0


Actually you need to implement per feature branch deployment. Because development environment use for testing app before QA.
And feature branch deployment is very difficult to implement especialy at the begginging of devops.

May be at start will be enough to use docker-compose to build latest version of developer's application. On their localhost.


Is Azure's confidential computing essentially encrypted RAM?

The article you are referencing mentions encrypted RAM but that was only a reference to possibilities of enhancing the technology more in the future, not about the current state of confidential computing.
Confidential computing means, that when data is in use these things are prevented:

Malicious insiders with administrative privilege or direct access to hardware on which it is being processed

Hackers and malware that exploit bugs in the operating system, application, or hypervisor

Third parties accessing it without their consent

according to Microsoft own announcement
So in other words, it is kind of equivalent to encryption and firewall that are seen in communication but this time applied to running programs in a Trusted Execution Environment (TEE), which the term for the area which is protected when server runs a program.


How to get server url from kub dashboard?

When running an EKS cluster, the best way to configure your ~/.kube/config is by using the SDK's update-kubeconfig:

aws eks update-kubeconfig --name your-cluster-name

Make sure to configure your credentials as AWS_* environment variables. Or even better use something like aws-vault.


Deploy Vm Vmware with simple Ansible playbook

https://github.com/ansible/ansible/issues/55551

The fix is included in Ansible 2.8.3 or higher. I had the 2.8.2 version


Ansible fails in Jenkins pipeline when run from docker

In my case it ended up that Jenkins was running docker agent with specific UID and GID. To get that fixed it required to rebuild that docker image with creating internal Jenkins user with the same UID and GID
For that purpose on top of the Jenkinsfile to crate that docker image I have added:
def user_id
def group_id
node {
   user_id = sh(returnStdout: true, script: 'id -u').trim()
   group_id = sh(returnStdout: true, script: 'id -g').trim()
}

and then during the build stage I have passed additional arguments to the docker as
--build-arg JenkinsUserId=${user_id} --build-arg JenkinsGroupId=${group_id}

then in the Dockerfile for that build:
FROM alpine:latest

#pick up provided ARGs for the bild

ARG JenkinsUserId
ARG JenkinsGroupId

//do your stuff here

#create Ansible config directory
RUN set -xe \
    && mkdir -p /etc/ansible

#create Ansible tmp directory
RUN set -xe \
    && mkdir -p /.ansible/tmp

#set ANSIBLE_LOCAL_TEMP
ENV ANSIBLE_LOCAL_TEMP /.ansible/tmp

#create Ansible cp directory
RUN set -xe \
    && mkdir -p /.ansible/cp

#set ANSIBLE_SSH_CONTROL_PATH_DIR
ENV ANSIBLE_SSH_CONTROL_PATH_DIR /.ansible/cp

# Create Jenkins group and user
RUN if ! id $JenkinsUserId; then \
    groupadd -g ${JenkinsGroupId} jenkins; \
    useradd jenkins -u ${JenkinsUserId} -g jenkins --shell /bin/bash --create-home; \
  else \
    addgroup --gid 1000 -S jenkins && adduser --uid 1000 -S jenkins -G jenkins; \
  fi

RUN addgroup jenkins root

# Tell docker that all future commands should run as the appuser user
USER jenkins

and finally update docker agent for the main pipeline which had issue:
   agent {
      docker {
         image 'artifactory.devbox.local/docker-local/myrepo/jdk8:latest'
         args '-v $HOME/.m2:/root/.m2 -v $HOME/.ssh:/home/jenkins/.ssh  -v /etc/ansible:/etc/ansible -v $HOME/.ansible/tmp:/.ansible/tmp -v $HOME/.ansible/cp:/.ansible/cp'
      }
   }


In what all cases would one use an external load balancer over the Ingress routing mesh in a Docker cluster?

You missed a point that there's still a need in balancing traffic before nodes in Docker cluster, and ELB is great for it (managed cluster, healthchecks, you don't pay for number of ELB instances per VPC, you can spawn spare nodes and register them in ELB automagically with Cloudwatch), great discussion on ServerFault. Basically, external balancer will split traffic between all nodes, while internal balancers will route it to nodes where service containers are running (which is where routing mesh is used).


The docker cache needs the following:

Build has been run on the same docker host previously (or you explicitly added flags to trust a pulled image) and
The previous image needs to still be on the build host (not pruned) and
The same previous layer and either
The same command being run (including the same environment/args) or
The same hash on the files being copied

If you see the cache being busted on a COPY or ADD command, and not the previous step, then you need to look at the hash being generated. All the files need to be identical, same file names, cases, file permissions, file owners, and the contents need to be bit for bit identical. You can look at the resulting checksum for this hash by running a docker image history command, e.g.:

$ docker image history golang:alpine
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
6b21b4c6e7a3        3 weeks ago         /bin/sh -c #(nop) WORKDIR /go                   0B
<missing>           3 weeks ago         /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOPATH=/go               0B
<missing>           3 weeks ago         /bin/sh -c set -eux;  apk add --no-cache --v…   344MB
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.12.7    0B
<missing>           3 weeks ago         /bin/sh -c [ ! -e /etc/nsswitch.conf ] && ec…   17B
<missing>           3 weeks ago         /bin/sh -c apk add --no-cache   ca-certifica…   551kB
<missing>           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:0eb5ea35741d23fe3…   5.58MB

You can include the --no-trunc option to get a full line. The key piece of data relevant to your question from the above image is in the ADD step where you see file:0eb5ea35741d23fe3…. If that checksum changes between two image builds, the cache will bust and you'll start a new set of layers.


What considerations should be taken into account for a good log file format?

If you are dealing with application logs, here are the three broad categories of information that a good log file should cover:

basic context (timestamp, log level, application name, source file name, source module/function, source line number, log message)
server context (host name, data center name unless host name has it encoded, cluster/pod info, container info)
user context (request URI, username or a hash, request ID etc)

If we are using JSON for logging, care must be taken to make sure that the keys are short enough to reduce the redundant overhead and also make it easier to query.  Using all lowercase and underscores for keys could be a better idea because some of the log aggregators like Splunk are case sensitive.

In the structured format, stack traces become a little tricky to handle.  One solution is to use a marker line to indicate the end of a long line rather than just the newline.


How to set a Circle CI environment variable for a project?

https://circleci.com/docs/2.0/env-vars/#setting-an-environment-variable-in-a-project

Seems "contexts" can be used but must then also be set to "reference" the context in the build yaml config file.

Instead to set a environment variable for a project:
Jobs > Cog / gear icon of a project > Build settings > Environment variable


How do I trigger a build pipeline on a TFVC branch commit to map and get that branch?

You've discovered one of the great pitfalls with TFVC. I have also not found an answer to this issue.

The only possible option I've seen around getting code from TFVC out of different branches is to base your pipeline off the highest point of the source tree possible (i.e. the highest common parent folder of all branches) and always build using source labels. You have to do this otherwise a label based build request will checkout nothing if the labeled path(s) do not fall under the pipeline's source root path.

A side effect of this is that in the build agent working directory the checked out code is going to land in a physical folder path that includes all the parent folders going up to the root specified by the pipeline's source root. If your build tasks are generic enough to tolerate this it might work. However, in my experience, the build tasks need to collect artifacts which will essentially be in an unknown location (based on the arbitrary branch) so you'll need something a bit smarter to get the path from which to collect said artifacts. It gets messy quickly.

Another side effect of this is that you can't do continuous integration builds. In order to support getting code from labeled locations, the pipeline's source location is the base path of all possible branches. A CI triggered build then has no context for which instance of a project to build (i.e. which branch). While you could probably extract the branch of a commit from the commit data, you'd need more complicated build scripts to do so.

I've found the pain of this to be far greater than just migrating to Git which allows for considerably more flexibility for building from arbitrary branches since the pipeline is based on the standard repo root from which all branches are based. Since Microsoft has done all but outright state that TFVC is going away, your time may be better invested in migrating to Git to take advantage of this and the other things it provides.


What are vertical scalability limits of JFrog Artifactory Pro?

As @XiongChiamiov mentioned, it's a very hard question to answer, but here's a clue:

Artifactory is a network-bond server. It means given enough storage, CPU and memory, it will max out your network interface first. Network interface is considered maxed out at half its rated bandwidth limitation. That can help you do the math.


Jenkins installed on aws unable to launch on browser

It seems your have not opened your 8080 port in Security Groups in AWS.

Steps:

Go to AWS console
Click Security Group and select the security group that you have assigned to your EC2 instance.
Click Edit Inbound Rules
Add a Custom TCP Rule and add 8080 in the Port Range.
You will now see Jenkins rise from the grave :) i.e. first time installation screen.

P.S.:
My Friend, if you have reached till Step 5, you know what to do next ;)


It looks like issue with ssh timeout is related to nested visualization restrictions. Thus, despite of enabling visualization in BIOS VM in container will still not be provisioned. I found similar answer here.

Once I started Docker container on host OS, issue has gone away and VM successfully provisioned.


Codeship cannot push image to Google Container Registry

The service account needs permissions to the storage segment where the containers will be pushed. To fix this

Log in to the Google Cloud Platform console
Go to the Storage section
Tick the corresponding storage segment and click Show information panel button in the top-right corner.
Add the storage object administration permission to your service account.

You can then retry locally with jet steps --push.


Best Practice Authenticating Jenkins to Bitbucket to Access Team Repository

In an enterprise environment (but even in a personal one, I assume tech savvy enough users to do that) it's generally a good idea to have dedicated accounts created specifically for automated tools execution.

I'd even recommend one account per tool because different tools may have different permission requirements and it's generally a good idea to limit the permission scope to the minimum required, for security reasons. In large enterprises this is often mandatory, prescribed by policies.

In some cases it might be necessary to have different tools executed under the same account, in which case such combo accounts would be required. Again, one per specific use case.

If you want a parallel - take a look at a Linux system, for example - different system accounts used for different purposes. Or service accounts, at Google for example.


Why will my container run in Docker, but will not run in Kubernetes?

To debug this, check the container logs and the container events.

kubectl get pods
Find your pod name.
kubectl logs
If logs don't make it obvious...
kubectl get events

If you deployed into a namespace, make sure to add -n  to all commands.

This should reveal basically everything. If you're stuck, you can also browse around the k8s dashboard if you haven't yet.  But that probably wouldn't reveal much more than the CLI is with these commands.


Do any services other than EC2s consume ip addresses in AWS?

There are many AWS services which requires a network interface such as AWS load balancers, Lambda, NAT gateway, VPC endpoints etc. For each AWS load balancer, it may use up to 2 IPs and 1 IP for 1 lambda function.
I would recommend to check the network interface section under EC2 service which will show you all the IPs which are currently being used and attached to EC2, Lambda, load balancer or some other service.
NOTE: For each subnet, 5 IP addresses will be reserved by AWS. So, if you have a subnet with /28 subnet mask (16 available IP addresses ), in AWS, you will get only 11 IPs which can be used.


How to clean up orphaned AWS EC2 snapshots?

Largely inspired by the blog posts and gist already linked in the other answers, here is my take to the problem.

I did use some convoluted JMESpath functions to get a list of snapshots and not require tr.

Disclaimer: Use at your own risks, I did my best to avoid any problem and keep sane defaults, but I won't take any blame if it cause problem to you.

#!/bin/sh
# remove x if you don't want to see the commands
set -ex

# Some variable initialisation with sane defaults
DRUN='--dry-run'
DO_DELETE=${1:-'no'}
REGION=${2:-'eu-west-1'}
ACCOUNTID=${3:-'self'}

# Get two temporary files
SNAP_FILE=$(mktemp)
IMAGE_FILE=$(mktemp)

# Get the snapshot list and the volume list
aws --region "$REGION" ec2 describe-snapshots --owner-ids "$ACCOUNTID" --query 'Snapshots[*].[SnapshotId]' --output text > "$SNAP_FILE"
aws --region "$REGION" ec2 describe-images --owners "$ACCOUNTID" --filters Name=state,Values=available --query 'Images[*].BlockDeviceMappings[*].Ebs.[SnapshotId]' --output text > "$IMAGE_FILE"

# Check if the outputed command should be dry-run (default) or not
if [ "$DO_DELETE" = "IAMSURE" ]
then
 DRUN=''
fi

# count each snapshot id, decrease when a volume reference it, print delete command for those with no volumes
awk -v REGION="$REGION" -v DRUN="$DRUN" '
FNR==NR { snap[$1]++; next } # increment snapshots and get to next line in file immediately

{ snap[$1]-- } # we changed file, decrease the snap counter when a volume reference it

END {
 for (s in snap) { # loop over the snapshots
   if (snap[s] > 0) { # if we did not decrese under 1 that means there is no volume referencing this snapshot
    cmd="aws --region " REGION " " DRUN " ec2 delete-snapshot --snapshot-id " s
    print(cmd)
  }
 }
}
' "$SNAP_FILE" "$IMAGE_FILE"
# Clean up the temp files
rm "$SNAP_FILE" "$IMAGE_FILE"

I hope the script itself is commented enough.

Default usage (no-params) will list delete commands of orphaned snapshots for the current account and region eu-west-1, extract:

aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-81e5856a
aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-95c68c7e
aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-a3bf50bd

You can redirect this output to a file for review before sourcing it to execute all the commands.

If you want the script to execute the command instead of printing them, replace print(cmd) by system(cmd).

Usage is as follow with a script named snap_cleaner:

for dry-run commands in us-west-1 region

./snap_cleaner no us-west-1

for usable commands in eu-central-1

./snap_cleaner IAMSURE eu-central-1

A third parameter can be used to access another account (I do prefer to switch role to another account before).

Stripped down version of the script with awk script as a oneliner:

#!/bin/sh
set -ex

# Some variable initialisation with sane defaults
DRUN='--dry-run'
DO_DELETE=${1:-'no'}
REGION=${2:-'eu-west-1'}
ACCOUNTID=${3:-'self'}

# Get two temporary files
SNAP_FILE=$(mktemp)
IMAGE_FILE=$(mktemp)

# Get the snapshot list and the volume list
aws --region "$REGION" ec2 describe-snapshots --owner-ids "$ACCOUNTID" --query 'Snapshots[*].[SnapshotId]' --output text > "$SNAP_FILE"
aws --region "$REGION" ec2 describe-images --owners "$ACCOUNTID" --filters Name=state,Values=available --query 'Images[*].BlockDeviceMappings[*].Ebs.[SnapshotId]' --output text > "$IMAGE_FILE"

# Check if the outputed command should be dry-run (default) or not
if [ "$DO_DELETE" = "IAMSURE" ]
then
 DRUN=''
fi

# count each snapshot id, decrease when a volume reference it, print delete command for those with no volumes
awk -v REGION="$REGION" -v DRUN="$DRUN" 'FNR==NR { snap[$1]++; next } { snap[$1]-- } END { for (s in snap) { if (snap[s] > 0) { cmd="aws --region " REGION " " DRUN " ec2 delete-snapshot --snapshot-id " s; print(cmd) } } }' "$SNAP_FILE" "$IMAGE_FILE"
# Clean up the temp files
rm "$SNAP_FILE" "$IMAGE_FILE"


How to create a host on a DigitalOcean team with Terraform?

According to the article you linked, Team Resources are shared. This means that all droplets (hosts) are shared across the team once you have a team in place.

source: https://www.digitalocean.com/community/tutorials/how-to-use-teams-on-digitalocean#team-resources

This means that each team member, using his own personal token, can manage all the resources shared with her. Terraform allows each user to use his own credentials when managing resources, and includes a state file (that must be shared as well). Managing shared tfstate and managing secrets is addressed in this DevOps.SE question.